csv2cwms

Note

Maintainers: Charles Graham

Use cwms-cli csv2cwms to load CSV time series data into CWMS Data API (CDA) with a user-defined JSON configuration file.

This page is the dedicated documentation entry for the command. For a complete config example, see Complete config example.

• For the JSON config reference, global defaults, and option table, start there.

• For installation and first-run setup, see Installation and Setup.

Overview

csv2cwms supports:

• mapping one or more CSV columns into CWMS time series

• optional expressions across CSV value columns (i.e. column4+column5 into TSID)

• configurable duplicate handling with use_if_multiple

• optional timestamp rounding with round_to_nearest

• optional timestamp-column selection with date_col

Setup and Run

For installation, dependency setup, and shared API arguments, see Installation and Setup.

For the JSON config structure itself, see Complete config example.

Config References

For csv2cwms, start from one of these:

• Complete config example

• Supported interval identifiers

The complete config page documents the JSON structure, global defaults, and supported per-file and per-timeseries keys.

If your source CSV contains commented lines, csv2cwms skips rows whose first non-whitespace character is # automatically.

Working Example

The repository includes sample csv2cwms test data that can be used as a working example.

Sample CSV input

"Time","Headwater","Tailwater","U1_MW","U1_MVAR","U2_MW","U2_Mvar","total_Pwr_discharge","Tot_Tainter_Opening","SG1"
03/25/2025 20:30:00,599.95,405.54,50.04,4.93,49.89,5.09,6700,0.00,21.00
03/25/2025 20:45:00,599.92,405.53,50.6,2.57,50.02,3.33,6772,0.00,21.00
03/25/2025 21:00:00,599.91,405.57,26.89,1.21,25.98,1.22,4027,0.00,21.00
03/25/2025 21:15:00,599.93,405.29,-0.01,-0.01,-0.06,-0.01,0,0.00,21.00
03/25/2025 21:30:00,599.95,405.54,50.04,4.93,49.89,5.09,6700,0.00,21.00

Sample config

Notice that not every column is used in this example.

{
    "interval": null,
    "use_if_multiple": "last",
    "input_files": {
        "BROK": {
            "data_path": "cwmscli/commands/csv2cwms/tests/data/sample_brok.csv",
            "store_rule": "REPLACE_ALL",
            "date_format": [
                "%m/%d/%Y %H:%M:%S",
                "%m/%d/%Y %H:%M"
            ],
            "timeseries": {
                "BROK.Elev.Inst.15Minutes.0.Rev-SCADA-cda": {
                    "columns": "Headwater",
                    "units": "ft",
                    "precision": 2
                },
                "BROK.Elev-Tailwater.Inst.15Minutes.0.Rev-SCADA-cda": {
                    "columns": "Tailwater",
                    "units": "ft",
                    "precision": 2
                },
                "BROK.Flow-Power.Inst.15Minutes.0.Rev-SCADA-cda": {
                    "columns": "total_Pwr_discharge",
                    "units": "cfs",
                    "precision": 0
                },
                "BROK.Power-Gen.Inst.15Minutes.0.Rev-SCADA-cda": {
                    "columns": "U1_MW+U2_MW",
                    "units": "MW",
                    "precision": 2
                },
                "BROK-Turbine1.Power-Gen.Inst.15Minutes.0.Rev-SCADA-cda": {
                    "columns": "U1_MW",
                    "units": "MW",
                    "precision": 2
                },
                "BROK-Turbine2.Power-Gen.Inst.15Minutes.0.Rev-SCADA-cda": {
                    "columns": "U2_MW",
                    "units": "MW",
                    "precision": 2
                }
            }
        }
    }
}

Example dry run

After installing cwms-cli and cwms-python, and after setting the shared API inputs described in Common API Arguments, you can run:

cwms-cli csv2cwms \
  --office SWT \
  --api-root https://cwms-data.usace.army.mil/cwms-data \
  --config cwmscli/commands/csv2cwms/tests/data/sample_config.json \
  --timezone America/Chicago \
  --dry-run

That example uses the sample config and sample CSV shipped in this repository and is a good starting point for verifying that parsing, mapping, and logging look correct before you point the command at a production config.

Common Issues

If you are having trouble running csv2cwms:

• Use the top-level cwms-cli --log-level option to increase logging detail. See Common API Arguments.

• Confirm the shared CDA API inputs are set correctly. See Common API Arguments.

• Confirm the JSON config matches the current input_files / data_path structure. See Complete config example.

• Verify the source CSV timestamps are in the timezone you are passing with --timezone.

Important Config Behavior

• By default, the timestamp is assumed to be in the first CSV column.

• Set date_col to the CSV header name when the timestamp is in a different column.

• CSV rows whose first non-whitespace character is # are ignored automatically.

• The default timezone is GMT.

• If round_to_nearest is enabled, config file interval takes precedence.

• If interval is not configured, rounding falls back to the interval parsed from the CWMS time series interval part. See Supported interval identifiers for the exact accepted values. Read the Time Series Docs for more on TSID structure.

• If multiple rows land in the same rounded timestamp, use_if_multiple controls whether the first, last, average, or an error is used. Error is the default behavior.

Timezone Handling

The --timezone option defaults to GMT.

Set --timezone when the timestamps in the CSV represent local clock time instead of GMT/UTC. This matters because csv2cwms uses that timezone when:

• parsing timestamp strings from the CSV

• converting parsed timestamps into epoch values

• rounding timestamps into interval buckets

If the CSV timestamps are local plant, office, or regional times and you leave the timezone at the default GMT, the stored times can be shifted by the wrong UTC offset. Set a real zone such as America/Chicago when the source CSV was produced in that local timezone.

Quality codes

csv2cwms currently emits a simple quality code for each generated point:

• 3 when the value is present

• 5 when the value is missing and the generated point is a gap

This is the behavior currently visible in debug output when the command logs the generated time series points. In other words, if debug logging shows a line like [TSID] 2025-03-25 12:00:00 -> 20.0 (quality: 3), that trailing integer is the quality code that will be written for that point.

Canonical Example

{
    "interval": 3600,
    "round_to_nearest": true,
    "use_if_multiple": "last",
    "input_files": {
        "BROK": {
            "data_path": "cwmscli/commands/csv2cwms/tests/data/sample_brok.csv",
            "store_rule": "REPLACE_ALL",
            "date_col": "Time",
            "date_format": [
                "%m/%d/%Y %H:%M:%S",
                "%m/%d/%Y %H:%M"
            ],
            "round_to_nearest": true,
            "use_if_multiple": "last",
            "timeseries": {
                "BROK.Elev.Inst.15Minutes.0.Rev-SCADA-cda": {
                    "columns": "Headwater",
                    "units": "ft",
                    "precision": 2
                }
            }
        }
    }
}

For a dedicated page showing only the canonical config example, see Complete config example.

CLI Reference

cwms-cli csv2cwms

Store CSV TimeSeries data to CWMS using a config file

Usage

cwms-cli csv2cwms [OPTIONS]

Options

-k, --api-key <api_key>

API key for CDA. Optional when a saved cwms-cli login token is available. Can also be provided by CDA_API_KEY.

-a, --api-root <api_root>

Required Api Root for CDA. Can be user defined or placed in a env variable CDA_API_ROOT

-o, --office <office>

Required Office to grab data for

--log-level <log_level>

Set logging verbosity (overrides default INFO).

Options:

DEBUG | INFO | WARNING | ERROR | CRITICAL

--input-keys <input_keys>

Input keys. Defaults to all keys/files with –input-keys=all. These are the keys under “input_files” in a given config file. This option lets you run a single file from a config that contains multiple files. Example: –input-keys=file1

Default:

'all'

-lb, --lookback <lookback>

Lookback period in HOURS

Default:

120

-v, --verbose

Verbose logging

-c, --config <config_path>

Required Path to JSON config file

--log <log>

Path to the log file.

Default:

Sentinel.UNSET

--dry-run

Log only (no HTTP calls)

--begin <begin>

YYYY-MM-DDTHH:MM (local to –tz)

-tz, --timezone <tz>

Default:

'GMT'

--ignore-ssl-errors

Ignore TLS errors (testing only)

--version

Show the version and exit.

Environment variables

CDA_API_KEY

Provide a default for -k

CDA_API_ROOT

Provide a default for -a

OFFICE

Provide a default for -o

LOG_LEVEL

Provide a default for --log-level

See also

• CLI reference

• Installation and Setup

• Common API Arguments

• Supported interval identifiers