processing.py#

Functions for processing converted SBS instrument data.

class seabirdscientific.processing.CastType(value)#: The subsection of data to use when splitting by upcast and/or downcast

class seabirdscientific.processing.MinVelocityType(value)#: The minimum velocity type used with loop edit DEPRECATED. Use Literal defined in loop_edit

class seabirdscientific.processing.WindowFilterType(value)#: The window filter type. See CDT Processing in the docs for details. DEPRECATED. Use literals defined in windowd_filter

seabirdscientific.processing.align_ctd(x: ndarray, offset: float, sample_interval: float, flag_value=-9.99e-29) → ndarray#

Takes an ndarray object of data for a single variable and applies a time offset to the series.

Performs interpolation when offset is not a factor of sampleinterval

Parameters:

x – array of values to apply shift to
offset – offset value to shift by (s)
sample_interval – time between samples (s)

Returns:

the aligned data

seabirdscientific.processing.bin_average(dataset: Dataset, bin_variable: str, bin_size: float, include_scan_count: bool = True, min_scans: int = 1, max_scans: int = 999999, exclude_bad_scans: bool = True, interpolate: bool = False, cast_type: CastType = CastType.BOTH, trim_start: int = 0, trim_end: int = 0, include_surface_bin: bool = False, surface_bin_min: float = 0, surface_bin_max: float = 5, surface_bin_value: float = 2.5, flag_value=-9.99e-29) → Dataset#

Averages data into bins, using intervals based on bin_variable. Returns a new dataframe with the binned data

Parameters:

dataset – Dataset containing all data to group into bins
bin_variable – The variable that will control binning, typically pressure, depth, time, or scan number. For scan number, use ‘nScan’.
bin_size – The bin width or range of data for each bin
include_scan_count – If True includes a column (nbin) in the returned dataframe for the number of scans in each bin. Defaults to True
min_scans – The minimum number of scans required in a bin for it to be included in the dataset. Defaults to 1
max_scans – the maximum number of scans allowed to for a bin to be included in the dataset. Defaults to 999999
exclude_bad_scans – If True, removes scans marked bad by loop edit (scans marked bad by wild edit are always excluded). Defaults to True
interpolate – If True interpolates bins after averaging. Defaults to False
cast_type – Sets which data to include. When binning by depth or pressure use, UPCAST, DOWNCAST, or BOTH. When binning by time or other variables use NA. Defaults to CastType.BOTH
trim_start – Remove this number of scans from the beginning of the initial dataset. Defaults to 0
trim_end – Remove this number of scans fro mteh end of the initial dataset. Defaults to 0
include_surface_bin – Includes a surface bin at the beginning of the downcast and/or at the end of the upcast. Defaults to False
surface_bin_min – The minimum value of the bin_variable to include in the surface bin. Defaults to 0 (value less than 0 will be set to 0)
surface_bin_max – The maximum value of the bin_variable to include in the surface bin. Defaults to 5 (this will be constrained to surface_bin_min and bin_size / 2)
surface_bin_value – The target value for interpolating the surface bin at the beginning of the downcast. Defaults to 2.5 (the target value for the upcast surface bin is always 0)
flag_value – The magical number indicating bad data. Defaults to -9.99e-29

Returns:

A new Dataset with binned data

seabirdscientific.processing.bouyancy_frequency(temp_conservative_subset: ndarray | None = None, salinity_abs_subset: ndarray | None = None, pressure_dbar_subset: ndarray | None = None, gravity: float = 0)#: Deprecated. Use conversion.buoyancy_frequency

seabirdscientific.processing.buoyancy(temperature_c: ndarray, salinity_prac: ndarray, pressure_dbar: ndarray, latitude: ndarray = 0, longitude: ndarray = 0, window_size: float = 11, use_modern_formula=True, flag_value=-9.99e-29)#: Deprecated. Use conversion.buoyancy

seabirdscientific.processing.butterworth_filter(x: ndarray, time_constant: float, sample_interval: float) → ndarray#

Applies a butterworth filter to a dataset.

Parameters:

x – A numpy array of floats
time_constant – 1 / (2 * pi * cutoff_frequency)
sample_interval – 1 / sampling_frequency

Returns:

the filtered data

seabirdscientific.processing.cell_thermal_mass(temperature: ndarray = [], conductivity: ndarray = [], amplitude: float = 1, time_constant: float = 1, sample_interval: float = 1, temperature_C: ndarray | None = None, conductivity_Sm: ndarray | None = None) → ndarray#

Removes conductivity cell thermal mass effects from measured conductivity.

Cell Thermal Mass uses a recursive filter to remove conductivity cell thermal mass effects from the measured conductivity [From the SeaSoft manual, page 92]

Parameters:

temperature_C – temperature in degrees C
conductivity_Sm – conductivity in S/m
amplitude – thermal anomaly amplitude (alpha)
time_constant – thermal anomoly time constant (1/beta)
sample_interval – time between samples

Returns:

the corrected conductivity in S/m

seabirdscientific.processing.loop_edit(measurand: ndarray, flag: ndarray, sample_interval: float, min_velocity_type: Literal['fixed', 'percent'] = 'fixed', min_velocity: float = 0.25, window_size: float = 300, mean_speed_percent: float = 20, remove_surface_soak: bool = True, min_soak_depth: float = 5, max_soak_depth: float = 20, use_deck_pressure_offset: bool = True, exclude_flags: bool = True, flag_value=-9.99e-29, latitude: float = 0, units: Literal['depth', 'pressure'] = 'depth') → ndarray#

Marks scans determined to be part of a pressure loop as bad.

Loop Edit marks scans bad by setting the flag value associated with the scan to badflag in data that has pressure slowdowns or reversals (typically caused by ship heave).

Parameters:

measurand – Salt water depth or pressure as an array of positive numbers
flag – Array of flag values. The flag for a good value is 0.0. The flag for a detected loop value defaults to -9.99e-29
sample_interval – Time interval between samples in seconds
min_velocity_type – Sets whether flags are based on min velocity or a percentage of mean speed
min_velocity – The minimum velocity for data to be considered good
window_size – Time interval to include in mean speed calculation
mean_speed_percent – Percentage of mean speed for data to be considered good
remove_surface_soak – If true, data that occur before the minimum soak depth is reached are marked as bad
min_soak_depth – The depth that must be reached before data can be considered good
max_soak_depth – The maximum depth that can be considered the start of a downcast
use_deck_pressure_offset – If true, min and max soak depths are offset by the first value in the depth array
exclude_flags – If true, existing bad flags are preserved
flag_value – Passing is 0.0, failing defaults to -9.99e-29.

Returns:: np.ndarray: the input data with updated flags

seabirdscientific.processing.loop_edit_depth(depth: ndarray, flag: ndarray, sample_interval: float, min_velocity_type: Literal['fixed', 'percent'], min_velocity: float, window_size: float, mean_speed_percent: float, remove_surface_soak: bool, min_soak_depth: float, max_soak_depth: float, use_deck_pressure_offset: bool, exclude_flags: bool, flag_value=-9.99e-29) → ndarray#: Deprecated. Use loop_edit

seabirdscientific.processing.loop_edit_pressure(pressure: ndarray, latitude: float, flag: ndarray, sample_interval: float, min_velocity_type: MinVelocityType, min_velocity: float, window_size: float, mean_speed_percent: float, remove_surface_soak: bool, min_soak_depth: float, max_soak_depth: float, use_deck_pressure_offset: bool, exclude_flags: bool, flag_value=-9.99e-29) → ndarray#: Deprecated. Use loop_edit

seabirdscientific.processing.low_pass_filter(x: ndarray, time_constant: float, sample_interval: float) → ndarray#

Applies a low pass filter as defined in the SeaSoft manual v7.26.8 page 101.

Parameters:

x – A numpy array of floats
time_constant – 1 / (2 * pi * cutoff_frequency)
sample_interval – 1 / sampling_frequency

Returns:

the filtered data

seabirdscientific.processing.split(dataset: Dataset, control_variable: str, cast_type=CastType.BOTH, exclude_bad_scans=False, min_value=-inf, drop=False) → Dataset#

Adds a cast_type coordinate to the dataset that labels samples as “upcast”, “downcast”, or “”

Parameters:

dataset – The dataset to add the cast_type to
control_variable – The variable to determine where to split, typically pressure or depth
split_mode – Determines which casts to include in the result, defaults to CastType.BOTH
exclude_bad_scans – If True, rows with bad flags will be excluded when determining the max depth or pressure. Defaults to False
min_value – Values below this will be excluded from the beginning of the downcast and end of the upcast

Returns:

A new dataset with the cast_type coordinate

seabirdscientific.processing.wild_edit(data: ndarray, flags: ndarray, std_pass_1: float, std_pass_2: float, scans_per_block: int, distance_to_mean: float, exclude_bad_flags: bool, flag_value=-9.99e-29) → ndarray#

Flags outliers in a dataset.

Outliers are flagged by iterating over the data in blocks, taking the mean and standard deviation, and flagging data outside the combined variance (standard deviation of the block multiplied by the standard deviation argument). Each block is processed three times: first to remove loop edit flags if applicable, second to temporarily remove outliers outside std_pass_1, third to remove outliers outside std_pass_2 from the returned data.

If the final block contains fewer samples than scans_per_block, data is backfilled from the previous block before computing the mean and standard deviation.

This algorithm may introduce nonlinearities in the data because it runs one block at a time instead of a rolling window. This is done to maintain parity with SBE Data Processing.

Parameters:

data – The data to be flagged, such as temperature or pressure
flags – Flag data from loop edit
std_pass_1 – Standard deviation for the first pass
std_pass_2 – Standard deviation for the second pass
scans_per_block – Number of samples to process in each block
distance_to_mean – Minimum threshod to flag data. Values within this range to mean are kept even if their standard deviation is outside the limit
exclude_bad_flags – Excludes all loop edit flags
flag_value – The flag value written in place of data. Defaults to -9.99e-29.

Returns:

The data with flag values in place of outliers

seabirdscientific.processing.window_filter(data_in: ndarray, flags: ndarray, window_type: Literal['boxcar', 'cosine', 'gaussian', 'median', 'triangle'], window_width: int, sample_interval: float, half_width=1, offset=0.0, exclude_flags=False, flag_value=-9.99e-29) → ndarray#

Filters a dataset by convolving it with an array of weights.

The available window filter types are boxcar, cosine, triangle, gaussian and median. Refer to the SeaSoft data processing manual version 7.26.8, page 108

Parameters:

data_in – data to be filtered
flags – flagged data defined by loop edit
window_type – the filter type (boxcar, cosine, triangle, or gaussian)
window_width – width of the window filter (must be odd)
sample_interval – sample interval of the dataset. Defaults to 1.0.
half_width – width of the guassian curve. Defaults to 1.
offset – shifts the center point of the gaussian. Defaults to 0.0.
exclude_flags – exclude values from the dataset that are flagged in flags. Also excludes corresponding weights from normalization. Defaults to False.
flag_value – the flag value in flags. Defaults to -9.99e-29.

Returns:

the convolution of data_in and the window filter