Skip to content

Commit 568703b

Browse files
edasmalchiedasmalchi
authored
Merge pull request #1735 from cal-itp/sep-oct-extra-dates
Sep oct extra dates
2 parents 770fdb7 + 73aab72 commit 568703b

19 files changed

+253
-17
lines changed

‎_shared_utils/shared_utils/rt_dates.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,12 @@
7272
"jun2025": "2025-06-11",
7373
"jul2025": "2025-07-16",
7474
"aug2025": "2025-08-20",
75+
"sep2025a": "2025-09-23",
7576
"sep2025": "2025-09-24",
77+
"sep2025b": "2025-09-25",
78+
"oct2025a": "2025-10-14",
79+
"oct2025": "2025-10-15",
80+
"oct2025b": "2025-10-16",
7681
}
7782

7883
years_available = list(range(2023, datetime.datetime.now().year + 1))

‎gtfs_funnel/Makefile‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ monthly_scheduled_data:
4141
python concatenate_monthly_scheduled_service.py
4242

4343
funnel_gtfs_single_day:
44-
make download_gtfs_data
44+
# make download_gtfs_data
4545
make preprocess_schedule_vp_dependency
4646
make preprocess_vp
4747
make preprocess_schedule_only
@@ -51,9 +51,9 @@ funnel_gtfs_single_day:
5151
all:
5252
make funnel_gtfs_single_day
5353
# update open_data/update_vars.py
54-
cd ../open_data/ && make create_gtfs_schedule_geospatial_open_data -f Makefile
54+
# cd ../open_data/ && make create_gtfs_schedule_geospatial_open_data -f Makefile
5555
# update high_quality_transit_areas/update_vars.py
56-
cd ../high_quality_transit_areas/ && make hqta_data -f Makefile
56+
# cd ../high_quality_transit_areas/ && make hqta_data -f Makefile
5757
# update rt_segment_speeds/segment_speed_utils/project_vars.py
5858
cd ../rt_segment_speeds/scripts/ && make all_speeds_pipeline -f Makefile && cd ../../
5959
# update rt_scheduled_v_ran/scripts/update_vars.py

‎gtfs_funnel/logs/download_data.log‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -282,3 +282,98 @@
282282
2025-10-10 11:11:47.561 | INFO | __main__:download_one_day:33 - # operators to run: 189
283283
2025-10-10 11:11:47.562 | INFO | __main__:download_one_day:37 - *********** Download st data ***********
284284
2025-10-10 11:13:38.047 | INFO | __main__:download_one_day:60 - execution time: 0:01:52.199603
285+
2025-10-19 13:56:28.825 | INFO | __main__:download_one_day:45 - Analysis date: 2025-09-23
286+
2025-10-19 13:56:30.610 | INFO | __main__:download_one_day:52 - # operators to run: 225
287+
2025-10-19 13:56:30.611 | INFO | __main__:download_one_day:56 - *********** Download trips data ***********
288+
2025-10-19 13:56:51.578 | INFO | __main__:download_one_day:86 - execution time: 0:00:22.752224
289+
2025-10-19 13:57:18.458 | INFO | __main__:download_one_day:22 - Analysis date: 2025-09-23
290+
2025-10-19 13:57:20.007 | INFO | __main__:download_one_day:29 - # operators to run: 225
291+
2025-10-19 13:57:20.007 | INFO | __main__:download_one_day:33 - *********** Download stops data ***********
292+
2025-10-19 13:57:28.876 | INFO | __main__:download_one_day:64 - execution time: 0:00:10.417604
293+
2025-10-19 13:57:55.267 | INFO | __main__:download_one_day:22 - Analysis date: 2025-09-23
294+
2025-10-19 13:57:56.675 | INFO | __main__:download_one_day:29 - # operators to run: 225
295+
2025-10-19 13:57:56.676 | INFO | __main__:download_one_day:33 - *********** Download routelines data ***********
296+
2025-10-19 13:59:00.479 | INFO | __main__:download_one_day:63 - execution time: 0:01:05.211390
297+
2025-10-19 13:59:27.765 | INFO | __main__:download_one_day:24 - Analysis date: 2025-09-23
298+
2025-10-19 13:59:31.079 | INFO | __main__:download_one_day:33 - # operators to run: 190
299+
2025-10-19 13:59:31.080 | INFO | __main__:download_one_day:37 - *********** Download st data ***********
300+
2025-10-19 14:00:30.237 | INFO | __main__:download_one_day:60 - execution time: 0:01:02.471711
301+
2025-10-19 14:32:18.048 | INFO | __main__:download_one_day:45 - Analysis date: 2025-09-25
302+
2025-10-19 14:32:19.500 | INFO | __main__:download_one_day:52 - # operators to run: 225
303+
2025-10-19 14:32:19.500 | INFO | __main__:download_one_day:56 - *********** Download trips data ***********
304+
2025-10-19 14:32:41.137 | INFO | __main__:download_one_day:86 - execution time: 0:00:23.089135
305+
2025-10-19 14:33:08.117 | INFO | __main__:download_one_day:22 - Analysis date: 2025-09-25
306+
2025-10-19 14:33:09.532 | INFO | __main__:download_one_day:29 - # operators to run: 225
307+
2025-10-19 14:33:09.532 | INFO | __main__:download_one_day:33 - *********** Download stops data ***********
308+
2025-10-19 14:33:23.210 | INFO | __main__:download_one_day:64 - execution time: 0:00:15.092713
309+
2025-10-19 14:33:49.972 | INFO | __main__:download_one_day:22 - Analysis date: 2025-09-25
310+
2025-10-19 14:33:51.414 | INFO | __main__:download_one_day:29 - # operators to run: 225
311+
2025-10-19 14:33:51.415 | INFO | __main__:download_one_day:33 - *********** Download routelines data ***********
312+
2025-10-19 14:34:54.998 | INFO | __main__:download_one_day:63 - execution time: 0:01:05.025703
313+
2025-10-19 14:35:21.800 | INFO | __main__:download_one_day:24 - Analysis date: 2025-09-25
314+
2025-10-19 14:35:25.009 | INFO | __main__:download_one_day:33 - # operators to run: 191
315+
2025-10-19 14:35:25.009 | INFO | __main__:download_one_day:37 - *********** Download st data ***********
316+
2025-10-19 14:36:21.872 | INFO | __main__:download_one_day:60 - execution time: 0:01:00.071617
317+
2025-10-19 15:19:39.754 | INFO | __main__:download_one_day:45 - Analysis date: 2025-10-14
318+
2025-10-19 15:19:41.476 | INFO | __main__:download_one_day:52 - # operators to run: 223
319+
2025-10-19 15:19:41.477 | INFO | __main__:download_one_day:56 - *********** Download trips data ***********
320+
2025-10-19 15:20:04.459 | INFO | __main__:download_one_day:86 - execution time: 0:00:24.704604
321+
2025-10-19 15:20:32.772 | INFO | __main__:download_one_day:22 - Analysis date: 2025-10-14
322+
2025-10-19 15:20:34.305 | INFO | __main__:download_one_day:29 - # operators to run: 223
323+
2025-10-19 15:20:34.306 | INFO | __main__:download_one_day:33 - *********** Download stops data ***********
324+
2025-10-19 15:20:43.480 | INFO | __main__:download_one_day:64 - execution time: 0:00:10.707514
325+
2025-10-19 15:21:11.552 | INFO | __main__:download_one_day:22 - Analysis date: 2025-10-14
326+
2025-10-19 15:21:13.003 | INFO | __main__:download_one_day:29 - # operators to run: 223
327+
2025-10-19 15:21:13.003 | INFO | __main__:download_one_day:33 - *********** Download routelines data ***********
328+
2025-10-19 15:22:18.230 | INFO | __main__:download_one_day:63 - execution time: 0:01:06.677572
329+
2025-10-19 15:22:47.258 | INFO | __main__:download_one_day:24 - Analysis date: 2025-10-14
330+
2025-10-19 15:22:50.815 | INFO | __main__:download_one_day:33 - # operators to run: 193
331+
2025-10-19 15:22:50.815 | INFO | __main__:download_one_day:37 - *********** Download st data ***********
332+
2025-10-19 15:23:52.117 | INFO | __main__:download_one_day:60 - execution time: 0:01:04.858397
333+
2025-10-19 15:35:26.641 | INFO | __main__:download_one_day:45 - Analysis date: 2025-10-15
334+
2025-10-19 15:35:28.116 | INFO | __main__:download_one_day:52 - # operators to run: 223
335+
2025-10-19 15:35:28.116 | INFO | __main__:download_one_day:56 - *********** Download trips data ***********
336+
2025-10-19 15:35:49.866 | INFO | __main__:download_one_day:86 - execution time: 0:00:23.225026
337+
2025-10-19 15:36:16.979 | INFO | __main__:download_one_day:22 - Analysis date: 2025-10-15
338+
2025-10-19 15:36:18.416 | INFO | __main__:download_one_day:29 - # operators to run: 223
339+
2025-10-19 15:36:18.416 | INFO | __main__:download_one_day:33 - *********** Download stops data ***********
340+
2025-10-19 15:36:27.232 | INFO | __main__:download_one_day:64 - execution time: 0:00:10.252833
341+
2025-10-19 15:36:53.624 | INFO | __main__:download_one_day:22 - Analysis date: 2025-10-15
342+
2025-10-19 15:36:54.954 | INFO | __main__:download_one_day:29 - # operators to run: 223
343+
2025-10-19 15:36:54.955 | INFO | __main__:download_one_day:33 - *********** Download routelines data ***********
344+
2025-10-19 15:38:02.018 | INFO | __main__:download_one_day:63 - execution time: 0:01:08.394299
345+
2025-10-19 15:38:31.402 | INFO | __main__:download_one_day:24 - Analysis date: 2025-10-15
346+
2025-10-19 15:38:34.750 | INFO | __main__:download_one_day:33 - # operators to run: 193
347+
2025-10-19 15:38:34.750 | INFO | __main__:download_one_day:37 - *********** Download st data ***********
348+
2025-10-19 15:39:32.814 | INFO | __main__:download_one_day:60 - execution time: 0:01:01.411832
349+
2025-10-19 16:01:46.179 | INFO | __main__:download_one_day:45 - Analysis date: 2025-10-15
350+
2025-10-19 16:01:47.587 | INFO | __main__:download_one_day:52 - # operators to run: 223
351+
2025-10-19 16:01:47.587 | INFO | __main__:download_one_day:56 - *********** Download trips data ***********
352+
2025-10-19 16:02:06.488 | INFO | __main__:download_one_day:86 - execution time: 0:00:20.309299
353+
2025-10-19 16:02:33.752 | INFO | __main__:download_one_day:22 - Analysis date: 2025-10-15
354+
2025-10-19 16:02:35.117 | INFO | __main__:download_one_day:29 - # operators to run: 223
355+
2025-10-19 16:02:35.117 | INFO | __main__:download_one_day:33 - *********** Download stops data ***********
356+
2025-10-19 16:02:42.132 | INFO | __main__:download_one_day:64 - execution time: 0:00:08.379071
357+
2025-10-19 16:03:08.466 | INFO | __main__:download_one_day:22 - Analysis date: 2025-10-15
358+
2025-10-19 16:03:09.803 | INFO | __main__:download_one_day:29 - # operators to run: 223
359+
2025-10-19 16:03:09.804 | INFO | __main__:download_one_day:33 - *********** Download routelines data ***********
360+
2025-10-19 16:04:05.893 | INFO | __main__:download_one_day:63 - execution time: 0:00:57.427203
361+
2025-10-19 16:04:33.529 | INFO | __main__:download_one_day:24 - Analysis date: 2025-10-15
362+
2025-10-19 16:04:37.218 | INFO | __main__:download_one_day:33 - # operators to run: 193
363+
2025-10-19 16:04:37.218 | INFO | __main__:download_one_day:37 - *********** Download st data ***********
364+
2025-10-19 16:05:28.851 | INFO | __main__:download_one_day:60 - execution time: 0:00:55.321191
365+
2025-10-19 16:24:18.407 | INFO | __main__:download_one_day:45 - Analysis date: 2025-10-16
366+
2025-10-19 16:24:19.917 | INFO | __main__:download_one_day:52 - # operators to run: 224
367+
2025-10-19 16:24:19.917 | INFO | __main__:download_one_day:56 - *********** Download trips data ***********
368+
2025-10-19 16:24:48.013 | INFO | __main__:download_one_day:86 - execution time: 0:00:29.605209
369+
2025-10-19 16:25:17.385 | INFO | __main__:download_one_day:22 - Analysis date: 2025-10-16
370+
2025-10-19 16:25:18.762 | INFO | __main__:download_one_day:29 - # operators to run: 224
371+
2025-10-19 16:25:18.763 | INFO | __main__:download_one_day:33 - *********** Download stops data ***********
372+
2025-10-19 16:25:27.897 | INFO | __main__:download_one_day:64 - execution time: 0:00:10.511597
373+
2025-10-19 16:25:59.398 | INFO | __main__:download_one_day:22 - Analysis date: 2025-10-16
374+
2025-10-19 16:26:00.974 | INFO | __main__:download_one_day:29 - # operators to run: 224
375+
2025-10-19 16:26:00.974 | INFO | __main__:download_one_day:33 - *********** Download routelines data ***********
376+
2025-10-19 16:27:07.062 | INFO | __main__:download_one_day:63 - execution time: 0:01:07.663294
377+
2025-10-19 16:27:35.377 | INFO | __main__:download_one_day:24 - Analysis date: 2025-10-16
378+
2025-10-19 16:27:38.674 | INFO | __main__:download_one_day:33 - # operators to run: 195
379+
2025-10-19 16:27:38.675 | INFO | __main__:download_one_day:37 - *********** Download st data ***********

‎gtfs_funnel/logs/download_vp_v2.log‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,3 +229,55 @@
229229
2025-10-10 11:23:51.390 | INFO | __main__:<module>:99 - Analysis date: 2025-08-20
230230
2025-10-10 11:23:59.344 | INFO | __main__:<module>:107 - concat and filter batched data: 0:00:07.952765
231231
2025-10-10 11:27:48.972 | INFO | __main__:<module>:114 - export concatenated vp: 0:03:49.628196
232+
2025-10-19 14:00:59.565 | INFO | __main__:<module>:172 - Analysis date: 2025-09-23
233+
2025-10-19 14:01:47.409 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 0 to GCS: 0:00:47.842476
234+
2025-10-19 14:02:20.544 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 1 to GCS: 0:00:33.134273
235+
2025-10-19 14:04:36.215 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 2 to GCS: 0:02:15.671178
236+
2025-10-19 14:05:12.850 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 3 to GCS: 0:00:36.633640
237+
2025-10-19 14:05:12.850 | INFO | __main__:<module>:179 - execution time: 0:04:13.283687
238+
2025-10-19 14:05:40.288 | INFO | __main__:<module>:99 - Analysis date: 2025-09-23
239+
2025-10-19 14:05:47.406 | INFO | __main__:<module>:107 - concat and filter batched data: 0:00:07.117668
240+
2025-10-19 14:08:17.446 | INFO | __main__:<module>:114 - export concatenated vp: 0:02:30.039175
241+
2025-10-19 14:10:02.513 | INFO | __main__:<module>:136 - remove batched parquets
242+
2025-10-19 14:10:02.517 | INFO | __main__:<module>:139 - execution time: 0:04:22.227845
243+
2025-10-19 14:36:50.114 | INFO | __main__:<module>:172 - Analysis date: 2025-09-25
244+
2025-10-19 14:37:37.676 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 0 to GCS: 0:00:47.559932
245+
2025-10-19 14:38:09.739 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 1 to GCS: 0:00:32.061858
246+
2025-10-19 14:40:48.352 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 2 to GCS: 0:02:38.613137
247+
2025-10-19 14:41:24.884 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 3 to GCS: 0:00:36.530217
248+
2025-10-19 14:41:24.884 | INFO | __main__:<module>:179 - execution time: 0:04:34.768308
249+
2025-10-19 14:41:55.575 | INFO | __main__:<module>:99 - Analysis date: 2025-09-25
250+
2025-10-19 14:42:02.700 | INFO | __main__:<module>:107 - concat and filter batched data: 0:00:07.124733
251+
2025-10-19 14:44:36.949 | INFO | __main__:<module>:114 - export concatenated vp: 0:02:34.249086
252+
2025-10-19 14:46:29.620 | INFO | __main__:<module>:136 - remove batched parquets
253+
2025-10-19 14:46:29.622 | INFO | __main__:<module>:139 - execution time: 0:04:34.047602
254+
2025-10-19 15:24:22.729 | INFO | __main__:<module>:172 - Analysis date: 2025-10-14
255+
2025-10-19 15:25:14.577 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 0 to GCS: 0:00:51.847630
256+
2025-10-19 15:25:46.025 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 1 to GCS: 0:00:31.448020
257+
2025-10-19 15:27:55.856 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 2 to GCS: 0:02:09.829720
258+
2025-10-19 15:28:36.093 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 3 to GCS: 0:00:40.236638
259+
2025-10-19 15:28:36.093 | INFO | __main__:<module>:179 - execution time: 0:04:13.363893
260+
2025-10-19 15:29:03.645 | INFO | __main__:<module>:99 - Analysis date: 2025-10-14
261+
2025-10-19 15:29:10.473 | INFO | __main__:<module>:107 - concat and filter batched data: 0:00:06.827539
262+
2025-10-19 15:31:43.045 | INFO | __main__:<module>:114 - export concatenated vp: 0:02:32.571956
263+
2025-10-19 15:33:30.983 | INFO | __main__:<module>:136 - remove batched parquets
264+
2025-10-19 15:33:30.986 | INFO | __main__:<module>:139 - execution time: 0:04:27.340553
265+
2025-10-19 15:40:01.115 | INFO | __main__:<module>:172 - Analysis date: 2025-10-15
266+
2025-10-19 15:40:53.352 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 0 to GCS: 0:00:52.236794
267+
2025-10-19 15:41:27.004 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 1 to GCS: 0:00:33.651457
268+
2025-10-19 15:43:34.285 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 2 to GCS: 0:02:07.280096
269+
2025-10-19 15:44:10.908 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 3 to GCS: 0:00:36.621913
270+
2025-10-19 15:44:10.908 | INFO | __main__:<module>:179 - execution time: 0:04:09.792488
271+
2025-10-19 15:44:37.898 | INFO | __main__:<module>:99 - Analysis date: 2025-10-15
272+
2025-10-19 15:44:44.821 | INFO | __main__:<module>:107 - concat and filter batched data: 0:00:06.922933
273+
2025-10-19 16:05:57.805 | INFO | __main__:<module>:172 - Analysis date: 2025-10-15
274+
2025-10-19 16:06:45.087 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 0 to GCS: 0:00:47.280298
275+
2025-10-19 16:07:14.948 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 1 to GCS: 0:00:29.861075
276+
2025-10-19 16:09:18.489 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 2 to GCS: 0:02:03.540730
277+
2025-10-19 16:09:52.936 | INFO | __main__:loop_through_batches_and_download_vp:135 - exported batch 3 to GCS: 0:00:34.445461
278+
2025-10-19 16:09:52.936 | INFO | __main__:<module>:179 - execution time: 0:03:55.129802
279+
2025-10-19 16:10:20.551 | INFO | __main__:<module>:99 - Analysis date: 2025-10-15
280+
2025-10-19 16:10:26.973 | INFO | __main__:<module>:107 - concat and filter batched data: 0:00:06.421744
281+
2025-10-19 16:12:51.534 | INFO | __main__:<module>:114 - export concatenated vp: 0:02:24.560596
282+
2025-10-19 16:14:47.639 | INFO | __main__:<module>:136 - remove batched parquets
283+
2025-10-19 16:14:47.642 | INFO | __main__:<module>:139 - execution time: 0:04:27.090519

‎gtfs_funnel/logs/preprocessing.log‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,6 @@
1111
2025-10-01 08:48:28.079 | INFO | __main__:assemble_stop_times_with_direction:274 - scheduled stop times with direction 2025-09-24: 0:07:30.478709
1212
2025-10-01 09:06:37.933 | INFO | __main__:assemble_stop_times_with_direction:274 - scheduled stop times with direction 2025-09-24: 0:07:26.845928
1313
2025-10-10 14:46:55.865 | INFO | __main__:assemble_stop_times_with_direction:274 - scheduled stop times with direction 2025-08-20: 0:07:46.742330
14+
2025-10-19 16:42:08.177 | INFO | __main__:assemble_stop_times_with_direction:274 - scheduled stop times with direction 2025-09-23: 0:07:52.345224
15+
2025-10-19 19:50:07.202 | INFO | __main__:assemble_stop_times_with_direction:274 - scheduled stop times with direction 2025-09-25: 0:08:13.800359
16+
2025-10-20 08:22:08.347 | INFO | __main__:assemble_stop_times_with_direction:274 - scheduled stop times with direction 2025-10-15: 0:07:55.271234

0 commit comments

Comments
 (0)