Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways
Yu Guo, Ryan Wen Liu^* , Jingxiang Qu, Yuxu Lu, Fenghua Zhu*, Yisheng Lv
(* Corresponding Author)
IEEE Transactions on Intelligent Transportation Systems

Introduction: The FVessel benchmark dataset is used to evaluate the reliability of AIS and video data fusion algorithms, which mainly contains 26 videos and the corresponding AIS data captured by the HIKVISION DS-2DC4423IW-D dome camera and Saiyang AIS9500-08 Class-B AIS receiver on the Wuhan Segment of the Yangtze River. To protect privacy, the MMSI for each vessel has been replaced with a random number in our dataset. As shown in Figure 1, these videos were captured under many locations (e.g., bridge region and riverside) and various weather conditions (e.g., sunny, cloudy, and low-light).

• 2024.07.20: New Website is created.
• 2024.04.03: FVessel dataset is included in the CVonline: Image Databases at the University of Edinburgh.
• 2023.08.01: 9 fusion data (Video-27~Video-35) have marked. When requesting the FVessel2.0, please contact us using your institutional or school email address exclusively for research purposes.
• 2023.06.08: "Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways" has been accepted by IEEE Transactions on Intelligent Transportation Systems.
• 2023.05.07: 9 fusion data (Video-27~Video-35) and 3728 images for detection are captured.
• 2023.01.12: We made the FVessel_V1.0 dataset public, containing 26 fusion data and 7625 images for detection.

The FVessel dataset consists of two parts:

• 01_Video AIS
• 02_Image xml

01_Video AIS contains many videos and the corresponding AIS data to evaluate the performance of the data fusion algorithm. Each video data contains the following files:

|-ais
     |-2022_05_10_19_21_04.csv
        |-[Number, MMSI, Lon, Lat, Speed, Course, Heading, Type, Timestamp]
        └─...
     |-2022_05_10_19_21_05.csv
     |-2022_05_10_19_21_06.csv
     └─... (ais data)
|-2022_05_10_19_21_05_19_31_10_b.mp4
|-camera_para.txt
   |-[Lon, Lat, Horizontal Orientation, Vertical Orientation, Camera Height, Horizontal FoV, Vertical FoV, fx, fy, u0, v0]
|-gt
     |-Video-02_gt.mp4
     |-Video-02_gt_detection.txt
        |-<second>, <0>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <x>, <y>, <z>
        └─...
     |-Video-02_gt_tracking.txt
        |-<second>, <id>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <x>, <y>, <z>
        └─...
     └─Video-02_gt_fusion.txt
        |-<second>, <mmsi>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <x>, <y>, <z>
        └─...

• (a) AIS

  Each csv file contains the AIS data received within ten minutes, and only the most recent data is kept.

  2022_05_10_19_21_04

  Number	MMSI	Lon	Lat	Speed	Course	Heading	Type	Timestamp
  0	100000000	114.325327	30.60166	0	293.6	511	18	1652181559844
  1	130000000	114.302683	30.58059	6.8	33.6	33	18	1652181659157
  2	140000000	114.31004	30.599997	3.9	215.6	511	18	1652181655147
  3	600000000	114.3156	30.59773	7.2	39.6	511	18	1652181649704
  ...	...	...	...	...	...	...	...	...

• (b) Video

  2022_05_10_19_21_05_19_31_10_b.mp4

  Starting time: 2022_05_10_19_21_05 End time: 2022_05_10_19_31_10 Type: b/r (bridge/riverside)

• (c) Camera Parameters

  camera_para.txt

  Lon	Lat	Horizontal Orientation	Vertical Orientation	Camera Height	Horizontal FoV	Vertical FoV	fx	fy	u0	v0
  114.32583	30.60139	7	-1	20	55	30.94	2391.26	2446.89	1305.04	855.214

  fx, fy, u0, and v0 are parameters in the internal matrix of the camera.

• (d) GT (Ground Truth: adopt the multi-object tracking MOT format)

  Video-02_gt.mp4

  Video containing ground truth, processed only once per second.

  Video-02_gt_detection.txt

  <second>, <0>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <x>, <y>, <z>
  

  Video-02_gt_tracking.txt

  <second>, <id>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <x>, <y>, <z>
  

  Video-02_gt_fusion.txt

  <second>, <mmsi>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <x>, <y>, <z>
  

• V1.0 (26 videos)

  Video	Video Length	Type	Weather	Times of Occlusions	Total Number of Vessels	Number of Vessels with AIS
  01	10m07s	Bridge	Low-light	2	5	4
  02	19m52s	Bridge	Sunny	6	7	6
  03	19m14s	Riverside	Sunny	6	5	5
  04	06m10s	Riverside	Sunny	0	1	1
  05	15m01s	Riverside	Sunny	2	5	5
  06	12m49s	Riverside	Sunny	2	4	4
  07	03m38s	Riverside	Sunny	1	2	2
  08	16m05s	Riverside	Sunny	3	6	5
  09	05m25s	Riverside	Sunny	0	1	1
  10	11m17s	Bridge	Cloudy	2	3	1
  11	05m18s	Riverside	Sunny	1	3	3
  12	07m19s	Riverside	Sunny	1	4	4
  13	12m58s	Riverside	Sunny	5	6	6
  14	03m58s	Riverside	Sunny	3	4	4
  15	10m46s	Riverside	Sunny	0	4	4
  16	05m05s	Riverside	Sunny	0	1	1
  17	08m08s	Riverside	Sunny	1	2	2
  18	23m57s	Riverside	Sunny	10	10	6
  19	11m28s	Riverside	Low-light	0	2	2
  20	14m10s	Riverside	Low-light	0	3	3
  21	24m01s	Riverside	Low-light	4	7	6
  22	02m40s	Riverside	Low-light	0	2	1
  23	19m24s	Riverside	Sunny	2	4	4
  24	08m39s	Riverside	Sunny	2	3	3
  25	24m05s	Riverside	Sunny	4	8	8
  26	07m26s	Riverside	Sunny	0	5	5

• V2.0 (9 videos)

02_Image xml contains many maritime images and the corresponding xml files for target detection network training. This dataset has only one class vessel.

|-JPEGImages
     |-000001.png
     |-000002.png
     |-000003.png
     └─... (image data)
|-Annotations
     |-000001.xml
        |-<vessel>, <x1>, <y1>, <x2>, <y2>
        └─...
     |-000002.xml
     |-000003.xml
     └─... (xml data)
└─-ImageSets

• V1.0 (7625 images)

FVessel_V1.0

FVessel_V2.0

Please contact us.

• Copy the data into the 01_demo_transform/data of this project.

• Run 01_demo_transformc/main.py.

Example: [mipc]

(The blue line is the projection of the AIS data-based trajectory in the image, and the red letter is the corresponding mmsi number.)

• Install motmetrics.

• Copy the two files from the 02_demo_metric/motmetrics of this project to the installed motmetrics folder.

• Save the test files to the 02_demo_metric/sample folder.

• Choose the type of evaluation detection, tracking, and fusion.

• Run 02_demo_metric/eval.py.

The following videos show the data fusion results of our proposed DeepSORVF.

We deeply thank Jianlong Su from the School of Computer and Artificial Intelligence in Wuhan University of Technology who performs the data acquisition and algorithm implementation works.

We will capture more data of different scenes to expand the dataset.

@article{guo2023asynchronous,
  title={Asynchronous trajectory matching-based multimodal maritime data fusion for vessel traffic surveillance in inland waterways},
  author={Guo, Yu and Liu, Ryan Wen and Qu, Jingxiang and Lu, Yuxu and Zhu, Fenghua and Lv, Yisheng},
  journal={IEEE Transactions on Intelligent Transportation Systems},
  volume={24},
  number={11},
  pages={12779--12792},
  year={2023}

Visitor count