Scientific Computing File Errors: Troubleshooting & Recovery Guide

Common Scientific Computing File Formats

Before diving into specific errors, it's important to understand the various file formats commonly used in scientific computing:

• HDF5 (.h5, .hdf5) - Hierarchical Data Format, a versatile format for storing large, complex datasets with rich metadata
• NetCDF (.nc, .cdf) - Network Common Data Form, widely used in climate science, geosciences, and atmospheric research
• MATLAB (.mat) - MATLAB's native format for storing workspace variables, widely used in engineering and signal processing
• Jupyter Notebooks (.ipynb) - JSON-based format that combines code, output, visualizations, and markdown documentation
• CSV/TSV (.csv, .tsv) - Simple tabular formats commonly used for data exchange
• Parquet/Arrow (.parquet, .arrow) - Columnar storage formats optimized for big data analytics
• FITS (.fits, .fit) - Flexible Image Transport System, standard in astronomy and astrophysics
• NPY/NPZ (.npy, .npz) - NumPy's binary format for storing array data efficiently
• Domain-specific formats - Formats like PDB (protein structures), GROMACS (molecular dynamics), or ROOT (particle physics)

Each format has specific structures, capabilities, and common issues. Understanding the format you're working with is crucial for effective troubleshooting.

Error #1: "HDF5 File Corrupted" or "Cannot Access HDF5 Dataset"

Symptoms

When attempting to open an HDF5 file, you may encounter error messages like "Unable to open HDF5 file," "HDF5 signature not found," or "Cannot read from dataset." Software may fail to load the file entirely, or it might load partially with missing datasets or groups.

Causes

• File truncation during transfer or storage
• Corrupted file headers or superblocks
• Interrupted write operations
• Storage media failures
• Incompatible HDF5 library versions
• Filesystem corruption
• Network issues during remote access

Solutions

Solution 1: Verify File Integrity with h5check

Use the HDF5 validation tools to identify issues:

1. Run h5check to validate the file structure:

   h5check filename.h5

2. Check for detailed error information about corruption location
3. For more information, use h5dump with error detection:

   h5dump -pH filename.h5

4. Review error codes and specific problem areas

Solution 2: Recover with h5repack or h5copy

Try to extract salvageable data:

1. Use h5repack to create a clean copy of the file:

   h5repack corrupted.h5 repaired.h5

2. If h5repack fails, try selective extraction with h5copy:

   h5copy -i corrupted.h5 -o extracted.h5 -s /path/to/dataset -d /path/to/dataset

3. For partially accessible files, selectively copy individual datasets or groups that are still readable

Solution 3: Programmatic Recovery using High-Level Libraries

Use programming libraries with error handling:

1. Python example using h5py with error handling:

   import h5py
   import numpy as np
   import traceback
   
   # Create a new file for recovered data
   recovered = h5py.File('recovered.h5', 'w')
   
   # Try to open the corrupted file with read-only and error handling
   try:
       with h5py.File('corrupted.h5', 'r', swmr=True) as f:
           # Function to recursively visit and try to copy groups/datasets
           def visit_and_recover(name, obj):
               try:
                   if isinstance(obj, h5py.Group):
                       # Create group in the recovered file if it doesn't exist
                       if name not in recovered:
                           recovered.create_group(name)
                       print(f"Successfully copied group: {name}")
                   elif isinstance(obj, h5py.Dataset):
                       # Try to read and copy the dataset
                       try:
                           data = obj[()]
                           # Recreate dataset in the recovered file
                           if name not in recovered:
                               recovered.create_dataset(name, data=data)
                           print(f"Successfully copied dataset: {name}")
                       except Exception as e:
                           print(f"Failed to recover dataset {name}: {str(e)}")
               except Exception as e:
                   print(f"Error processing {name}: {str(e)}")
                   
           # Visit all objects in the file
           f.visititems(visit_and_recover)
           
   except Exception as e:
       print(f"Failed to open file: {str(e)}")
       traceback.print_exc()
       
   finally:
       # Always close the recovered file
       recovered.close()
       print("Recovery attempt completed.")

Solution 4: Use Low-Level HDF5 Recovery Tools

For more severe corruption, try specialized approaches:

1. Check if the HDFGroup's recovery tools are applicable to your case:
   • h5recover for superblock damage
   • h5repair for selective block recovery
2. For corrupted metadata but intact raw data, consider byte-level extraction tools
3. Commercial data recovery services specializing in scientific formats may be able to help with severe corruption

Solution 5: Preventive Replication for Critical HDF5 Files

To avoid future data loss, implement protective measures:

1. Use h5repack periodically to clean and optimize important files:

   h5repack -f GZIP=9 original.h5 optimized.h5

2. Implement checksumming for datasets:

   h5repack -f FLETCHER32 original.h5 checksummed.h5

3. Consider storing critical data with redundancy using mirrored HDF5 files

Error #2: "NetCDF Read Error" or "Invalid Dimensions"

Symptoms

When working with NetCDF files, you may encounter errors like "NetCDF: Invalid dimensions," "NetCDF: Not a valid file format," or "Error accessing variable." Parts of the file may be inaccessible, or dimensional information may be inconsistent.

Causes

• Incomplete file transfers
• File header corruption
• Version incompatibilities (NetCDF-3 vs. NetCDF-4)
• Dimension or variable name corruption
• Conflicts between dimensional definitions
• Incorrect attribute types or values
• Storage or networking issues during write operations

Solutions

Solution 1: NetCDF File Validation and Analysis

Analyze the file structure to identify issues:

1. Use ncdump to examine the file structure:

   ncdump -h filename.nc

2. For more detailed checking, use nccheck or nc-verify tools
3. Check for specific error information pointing to corrupted sections
4. Verify version compatibility:

   ncdump -k filename.nc

Solution 2: Convert Between NetCDF Versions

Address version incompatibility issues:

1. Convert NetCDF-4 to NetCDF-3:

   ncks -3 input.nc output.nc

2. Convert NetCDF-3 to NetCDF-4:

   ncks -4 input.nc output.nc

3. Try conversion with compression for optimized storage:

   ncks -4 -L 4 input.nc compressed.nc

4. For specific file format issues, try forcing a format type:

   ncks --fl_fmt=netcdf4_classic input.nc output.nc

Solution 3: Extract Variables and Rebuild the File

Salvage individual components from the damaged file:

1. Use NCO tools to extract variables selectively:

   ncks -v variable_name input.nc extracted_var.nc

2. Extract dimension information and attributes:

   ncks -v .dimension_name input.nc extracted_dim.nc

3. Merge salvaged components into a new file:

   ncks -A extracted_var1.nc new.nc
   ncks -A extracted_var2.nc new.nc

Solution 4: Programmatic NetCDF Repair with Python

Use the netCDF4 library for controlled file repair:

1. Python example for selective recovery:

   import netCDF4 as nc
   import numpy as np
   
   # Open a new file for recovered data
   recovered = nc.Dataset('recovered.nc', 'w')
   
   try:
       # Try to open the corrupted file in read-only mode
       with nc.Dataset('corrupted.nc', 'r') as src:
           # Copy dimensions
           for dim_name, dimension in src.dimensions.items():
               try:
                   recovered.createDimension(dim_name, len(dimension) if not dimension.isunlimited() else None)
                   print(f"Copied dimension: {dim_name}")
               except Exception as e:
                   print(f"Failed to copy dimension {dim_name}: {str(e)}")
           
           # Copy global attributes
           for attr_name in src.ncattrs():
               try:
                   recovered.setncattr(attr_name, src.getncattr(attr_name))
                   print(f"Copied global attribute: {attr_name}")
               except Exception as e:
                   print(f"Failed to copy global attribute {attr_name}: {str(e)}")
           
           # Copy variables
           for var_name, variable in src.variables.items():
               try:
                   # Create the variable in the new file
                   var_type = variable.datatype
                   var_dims = variable.dimensions
                   var_out = recovered.createVariable(var_name, var_type, var_dims)
                   
                   # Copy variable attributes
                   for attr_name in variable.ncattrs():
                       var_out.setncattr(attr_name, variable.getncattr(attr_name))
                   
                   # Copy the data
                   var_out[:] = variable[:]
                   print(f"Copied variable: {var_name}")
               except Exception as e:
                   print(f"Failed to copy variable {var_name}: {str(e)}")
                   
   except Exception as e:
       print(f"Error opening corrupted file: {str(e)}")
       
   finally:
       # Close the recovered file
       recovered.close()
       print("Recovery attempt completed.")

Solution 5: CDO and NCO Tools for Advanced Repair

Leverage climate data operators for recovery:

1. Use CDO to fix common NetCDF issues:

   cdo copy input.nc fixed.nc

2. Repair time dimension issues:

   cdo settaxis,yyyy-mm-dd,hh:mm:ss,timeunit input.nc fixed.nc

3. Fix grid definition problems:

   cdo setgrid,gridfile.txt input.nc fixed.nc

4. Try selective data extraction and concatenation for corrupted timeseries:

   cdo seldate,yyyy-mm-dd,yyyy-mm-dd input.nc part1.nc
   cdo seldate,yyyy-mm-dd,yyyy-mm-dd input.nc part2.nc
   cdo mergetime part1.nc part2.nc merged.nc

Error #3: "MATLAB File Format Error" or "MAT-File Variable Import"

Symptoms

When trying to load MATLAB (.mat) files, you may see error messages like "Invalid MAT-file," "Unable to read MAT-file header," or "Error reading variable from file." Variables may be missing, corrupted, or have incorrect types when loaded.

Causes

• Version incompatibilities (MATLAB 5.0 vs. 7.3 formats)
• Corrupted file headers
• Partial file saves due to crashes
• 64-bit vs. 32-bit data storage issues
• Platform-specific data format differences
• Compression errors in newer MAT formats
• Mixed version saves from different MATLAB versions

Solutions

Solution 1: Try Different MATLAB Loading Options

Adjust loading parameters to accommodate corruption:

1. In MATLAB, use the 'load' command with options:

   % Try different MATLAB versions' formats
   try
       % Try v7.3 format (HDF5-based)
       data = load('corrupt.mat', '-mat', '-v7.3');
   catch
       try
           % Try v7 format
           data = load('corrupt.mat', '-mat', '-v7');
       catch
           try
               % Try v6 format (MATLAB 5.0)
               data = load('corrupt.mat', '-mat', '-v6');
           catch
               error('All loading attempts failed');
           end
       end
   end

2. Try loading variables selectively to isolate corruption:

   % List what variables are in the file
   vars = who('-file', 'corrupt.mat');
   
   % Try loading each variable separately
   for i = 1:length(vars)
       try
           var_data = load('corrupt.mat', vars{i});
           fprintf('Successfully loaded: %s\n', vars{i});
       catch
           fprintf('Failed to load: %s\n', vars{i});
       end
   end

Solution 2: Convert MAT File Versions

Transform between different MATLAB formats:

1. Load and re-save in a different format:

   % Load whatever can be loaded
   try
       data = load('corrupt.mat');
       
       % Save in older format which might be more robust
       save('recovered_v6.mat', '-struct', 'data', '-v6');
       
       % Or save in newer format
       save('recovered_v7.mat', '-struct', 'data', '-v7');
   catch e
       fprintf('Error during conversion: %s\n', e.message);
   end

2. For large files that might be using v7.3 (HDF5-based), try HDF5 tools:

   % Use low-level HDF5 functions to access 7.3 format files
   fileinfo = h5info('corrupt.mat');
   datasets = {fileinfo.Datasets.Name};
   
   % Extract datasets one by one
   for i = 1:length(datasets)
       try
           data.(datasets{i}) = h5read('corrupt.mat', ['/' datasets{i}]);
           fprintf('Successfully extracted dataset: %s\n', datasets{i});
       catch
           fprintf('Failed to extract dataset: %s\n', datasets{i});
       end
   end
   
   % Save recovered data
   save('recovered.mat', '-struct', 'data');

Solution 3: Use Third-Party Tools for MAT File Recovery

Leverage alternative libraries for loading MATLAB files:

1. Python example using scipy.io:

   import scipy.io as sio
   import h5py
   import numpy as np
   
   # Try loading with scipy
   try:
       data = sio.loadmat('corrupt.mat')
       print("Successfully loaded with scipy.io")
       # Save back to a new mat file
       sio.savemat('recovered_scipy.mat', data)
   except Exception as e:
       print(f"scipy.io failed: {str(e)}")
       
       # Try HDF5 approach for v7.3 files
       try:
           with h5py.File('corrupt.mat', 'r') as f:
               # Create a dictionary to hold the data
               data = {}
               
               # Function to recursively visit all objects
               def visit_and_extract(name, obj):
                   if isinstance(obj, h5py.Dataset):
                       try:
                           # Convert to numpy array
                           data[name] = np.array(obj)
                           print(f"Extracted: {name}")
                       except Exception as e:
                           print(f"Failed to extract {name}: {str(e)}")
               
               # Visit all objects
               f.visititems(visit_and_extract)
               
               # Save recovered data with scipy
               if data:
                   sio.savemat('recovered_h5py.mat', data)
                   print("Saved recovered data")
       except Exception as e:
           print(f"HDF5 approach failed: {str(e)}")

Solution 4: Binary Analysis for Header Repair

For advanced users, fix file headers manually:

1. MATLAB MAT files have specific header structures depending on version:
   • MAT 5.0 format starts with a 128-byte header
   • The first 4 bytes should be 'MATLAB'
2. Use a hex editor to verify and potentially fix simple header corruption
3. For v7.3 files, use HDF5 header repair tools since they use HDF5 format

Solution 5: Partial Reconstruction from Research Results

When direct recovery fails, reconstruct critical data:

1. Check for exported figures or data that might contain the essential information
2. Look for script files that generated the data originally
3. Check for derivative files or analysis results that might contain copies of variables
4. If source data for calculations is available, rerun analyses to regenerate results

Error #4: "Jupyter Notebook Parse Error" or "Invalid Notebook Format"

Symptoms

When opening a Jupyter notebook (.ipynb file), you may encounter errors like "Notebook validation failed," "Invalid JSON," or "Unable to parse notebook." JupyterLab or Jupyter Notebook may fail to load the file, or display a corrupted version with missing cells or content.

Causes

• Corrupted JSON structure
• Interrupted save operations during kernel activity
• Notebook server crashes during autosave
• Merge conflicts in version control systems
• Manual edits to the notebook file
• JupyterLab/Notebook version incompatibilities
• Extremely large output cells causing parsing issues

Solutions

Solution 1: Jupyter Notebook Format Validation and Repair

Check and fix JSON structure issues:

1. Use the nbformat validation tool:

   jupyter nbconvert --to notebook --validate corrupted.ipynb

2. For more detailed diagnostics:

   python -m nbformat.validator corrupted.ipynb

3. Try the notebook repair extension if available:

   pip install nbrepair  # If available
   jupyter nbrepair corrupted.ipynb

Solution 2: Fix JSON Structure Manually

Address specific JSON formatting issues:

1. Open the .ipynb file in a text editor (it's just JSON)
2. Look for obvious JSON errors:
   • Missing or extra commas
   • Unclosed brackets or braces
   • Incomplete string values (missing quote marks)
3. Use an online JSON validator to identify specific syntax errors
4. Focus on fixing structural issues rather than content initially

Solution 3: Extract Cells and Content Programmatically

Recover individual notebook components:

1. Python script to extract salvageable cells:

   import json
   import nbformat
   
   # Try to open the corrupted notebook
   try:
       with open('corrupted.ipynb', 'r', encoding='utf-8') as f:
           content = f.read()
       
       # Try to parse the JSON, even if it's partially corrupted
       notebook_data = json.loads(content)
       
       # Extract cells
       cells = []
       if 'cells' in notebook_data:
           for i, cell in enumerate(notebook_data['cells']):
               try:
                   # Validate each cell
                   if 'cell_type' in cell and 'source' in cell:
                       cells.append(cell)
                       print(f"Successfully extracted cell {i}")
                   else:
                       print(f"Skipping cell {i} due to missing required fields")
               except Exception as e:
                   print(f"Error processing cell {i}: {str(e)}")
       
       # Create a new notebook with the salvageable cells
       new_notebook = nbformat.v4.new_notebook()
       new_notebook.cells = cells
       
       # If metadata is available, try to preserve it
       if 'metadata' in notebook_data:
           try:
               new_notebook.metadata = notebook_data['metadata']
           except:
               print("Could not recover metadata")
       
       # Write the repaired notebook
       with open('recovered.ipynb', 'w', encoding='utf-8') as f:
           nbformat.write(new_notebook, f)
       
       print(f"Recovered {len(cells)} cells to recovered.ipynb")
       
   except Exception as e:
       print(f"Failed to recover notebook: {str(e)}")
       
       # If JSON parsing completely fails, try to extract content with regex
       import re
       try:
           with open('corrupted.ipynb', 'r', encoding='utf-8') as f:
               content = f.read()
           
           # Extract code blocks
           code_blocks = re.findall(r'"source":\s*\[(.*?)\]', content, re.DOTALL)
           
           # Create a simple text file with extracted code
           with open('extracted_code.txt', 'w', encoding='utf-8') as f:
               for i, block in enumerate(code_blocks):
                   f.write(f"--- BLOCK {i} ---\n")
                   # Remove JSON formatting
                   cleaned = re.sub(r'",\s*"', '\n', block)
                   cleaned = re.sub(r'"', '', cleaned)
                   # Unescape newlines
                   cleaned = cleaned.replace('\\n', '\n')
                   f.write(cleaned)
                   f.write('\n\n')
           
           print(f"Extracted {len(code_blocks)} code blocks to extracted_code.txt")
       
       except Exception as e2:
           print(f"Even basic content extraction failed: {str(e2)}")

Solution 4: Recover from Jupyter Autosave or Checkpoints

Look for automatic backups created by Jupyter:

1. Check for checkpoint files in the .ipynb_checkpoints directory:

   ls -la .ipynb_checkpoints/

2. Restore from the checkpoint version:

   cp .ipynb_checkpoints/notebook_name-checkpoint.ipynb recovered.ipynb

3. For JupyterLab, look for autosave files with names like:

   ls -la ~/.jupyter/lab/workspaces/

Solution 5: Convert to Other Formats and Rebuild

Try conversion to simpler formats:

1. If the notebook partially opens, export to a different format:

   jupyter nbconvert --to python corrupted.ipynb

2. For markdown content:

   jupyter nbconvert --to markdown corrupted.ipynb

3. Create a new notebook and copy salvageable content from these exports
4. If output data is critical, try extracting just the HTML:

   jupyter nbconvert --to html corrupted.ipynb

Error #5: "NumPy Array Loading Error" or "NPY Format Issue"

Symptoms

When trying to load NumPy binary files (.npy, .npz), you may encounter errors like "Unable to read array header," "Invalid NPY format," or "Cannot load NPZ file." The data may fail to load entirely, or load with incorrect shapes or data types.

Causes

• Corrupted file headers
• Incompatible NumPy versions
• Endianness issues across different platforms
• Incomplete file writes
• Mixed data type corruption
• Compression errors in NPZ files

Solutions

Solution 1: NumPy Loading with Error Handling

Try different loading approaches:

1. Python code with flexible loading options:

   import numpy as np
   
   def try_load_npy(filename):
       # Try different approaches to load a potentially corrupted NPY file
       try:
           # Standard approach
           data = np.load(filename)
           print("Standard loading successful")
           return data
       except Exception as e1:
           print(f"Standard loading failed: {str(e1)}")
           
           try:
               # Try with allow_pickle
               data = np.load(filename, allow_pickle=True)
               print("Loading with allow_pickle successful")
               return data
           except Exception as e2:
               print(f"allow_pickle loading failed: {str(e2)}")
               
               try:
                   # Try with fixing
                   data = np.load(filename, allow_pickle=True, fix_imports=True)
                   print("Loading with fix_imports successful")
                   return data
               except Exception as e3:
                   print(f"fix_imports loading failed: {str(e3)}")
                   
                   try:
                       # Try with mmap_mode for large files
                       data = np.load(filename, mmap_mode='r')
                       print("Loading with mmap_mode successful")
                       return data
                   except Exception as e4:
                       print(f"mmap_mode loading failed: {str(e4)}")
                       
                       # All attempts failed
                       print("All loading attempts failed")
                       return None
   
   # For NPZ files
   def try_load_npz(filename):
       try:
           # Standard approach
           data = np.load(filename)
           print("NPZ loading successful")
           print(f"Available arrays: {list(data.keys())}")
           return data
       except Exception as e:
           print(f"NPZ loading failed: {str(e)}")
           
           # Try opening as a zip file
           try:
               import zipfile
               with zipfile.ZipFile(filename) as z:
                   print(f"NPZ file contains: {z.namelist()}")
                   # Extract individual arrays
                   arrays = {}
                   for name in z.namelist():
                       if name.endswith('.npy'):
                           try:
                               with z.open(name) as f:
                                   # Read the file into a BytesIO object
                                   import io
                                   data_bytes = io.BytesIO(f.read())
                                   # Try to load the array
                                   arr = np.load(data_bytes)
                                   arrays[name[:-4]] = arr  # Remove .npy extension
                                   print(f"Successfully extracted array: {name}")
                           except Exception as e2:
                               print(f"Failed to extract {name}: {str(e2)}")
                   return arrays
           except Exception as e3:
               print(f"Zip extraction failed: {str(e3)}")
               return None

Solution 2: Repair NumPy File Headers

Fix header information in corrupted files:

1. Understanding the NPY format:
   • NPY files start with a magic string ('\x93NUMPY')
   • Followed by version byte, header length, and descriptor
2. Create a script to fix common header issues:

   import numpy as np
   import struct
   
   def repair_npy_header(corrupted_file, repaired_file, expected_shape, dtype):
       """
       Attempt to repair a corrupted NPY file by reconstructing its header
       
       Parameters:
       corrupted_file - Path to the corrupted NPY file
       repaired_file - Where to save the repaired file
       expected_shape - Tuple with the expected array shape
       dtype - Expected data type (e.g., 'float32', 'int64')
       """
       try:
           # Read the raw data from the corrupted file
           with open(corrupted_file, 'rb') as f:
               content = f.read()
           
           # Check if the magic string is present
           if not content.startswith(b'\x93NUMPY'):
               print("Magic string missing, adding NPY header")
               
               # Determine the size of the data
               dtype_obj = np.dtype(dtype)
               header = {
                   'descr': dtype_obj.str,
                   'fortran_order': False,
                   'shape': expected_shape
               }
               
               # Convert header to string representation
               header_str = repr(header).replace("'", '"')
               # Pad for 16-byte alignment
               header_bytes = header_str.encode('utf-8')
               padding = 16 - ((len(header_bytes) + 10) % 16)
               header_bytes = header_bytes + b' ' * padding + b'\n'
               # Format: 6-byte magic string + 4-byte header length + header
               magic = b'\x93NUMPY'
               version = struct.pack('BB', 1, 0)
               header_len = struct.pack('