[Spain] Add GTFS feeds from NAP

[Robot8A]

The Spanish NAP (National Access Point) is available here https://nap.mitma.es/. It has feeds from operator companies and regional governments mixed, so some preprocessing to avoid duplicates may be needed. The NAP only has the feeds from the companies and governments who want to release them, having therefore gaps in the coverage.

[Chaikney]

I am having a quick look at this now. The NAP page shows 114 data sources and the es.json file appears to have 110 sources. (Crude count on "name" excluding the 2 maintainers.)
Before I go deeper into trying to find what the other 4 are, are there criteria that might be excluding some of those sources? (I thought it might exclude the ferries / maritimo, but the ferries are there.)

[jbruechert]

I think they might not have existed when the list was put together. If you can find a machine readable list from the NAP, it would be great to have a script to generate feeds/es.json, like we have for France and Austria.

[Robot8A]

There is an API, which documentation is here: https://nap.transportes.gob.es/Account/InstruccionesAPI

[Altonss]

Ouigo ES is currently in es.json but seems not working 🤔 Should I open a separate issue?
Iryo seems also to be missing currently :)

[Chaikney]

"Ouigo ES is currently in es.json but seems not working 🤔 Should I open a separate issue?"
The Ouigo file is not getting fetched correctly, or the data it returns is bad?
"Iryo seems also to be missing currently :)"
Iryo is not in that NAP list, true. I wonder if we wait for NAP to add it (Iryo are a new provider I think) or if we can track down their data first...

" it would be great to have a script to generate feeds/es.json, like we have for France and Austria."
Agreed this would be the optimal solution.
Between the API and the example scripts I might be able to hack something together but I am short on time and experience, so I am not committing to producing anything usable.

[Altonss]

The Ouigo file is not getting fetched correctly, or the data it returns is bad?

I don't know what is the exact issue, but no train between Madrid and Barcelona is shown...

[GerdC]

FGC realtime data https://dadesobertes.fgc.cat/explore/?sort=modified&refine.keyword=gtfs+realtime

[Altonss]

Is it possible that URLs change over time, for example we should have AECFA Slots, but nothing appears in Transitous : https://nap.transportes.gob.es/Files/Detail/920 ?

[jbruechert]

Is it possible that URLs change over time, for example we should have AECFA Slots, but nothing appears in Transitous : https://nap.transportes.gob.es/Files/Detail/920 ?

It would probably be good to be able to generate the list from the NAP automatically to catch such cases, like we already do for France.

[Altonss]

Is it possible that URLs change over time, for example we should have AECFA Slots, but nothing appears in Transitous : https://nap.transportes.gob.es/Files/Detail/920 ?

The file url used by Transitous seems to be fine, and the data correct too. Any idea on why the flights do not show up on the map and routing?

[jbruechert]

I can't check it on the phone right now, but the first step is too see whether the post-processed feed (https://routing.spline.de/gtfs/es_AECFA-slots.gtfs.zip) looks reasonable, and if not, what the import log on the CI says.

[jbruechert]

The stop time syntax is invalid in the feed, which causes no data to actually be imported.
See https://gtfs-validator-results.mobilitydata.org/b9479fe1-0ae8-423c-acd8-a07580fcb54a/report.html

[Altonss]

The stop time syntax is invalid in the feed, which causes no data to actually be imported. See https://gtfs-validator-results.mobilitydata.org/b9479fe1-0ae8-423c-acd8-a07580fcb54a/report.html

The CI/gtfsclean didn't complain about it? Should we add a fix to accept/convert from HH:MM?

[jbruechert]

Currently all the entries in es.json have "fix": true, because manually looking through them was too much work for that number of feeds.
The problem is that the option will just delete all entries that it can't fix automatically, and apparently this issue is one of them. That's exactly the reason while "fix": true is not the default, because it's easy not to notice that this is happening.

I think it would be fine to extend gtfsclean to handle this, but I'd try to contact the feed producer first, since its much easier on their side and every consumer of the feed will need this fix.

[felixguendling]

The time parsing only reads HH:MM anyway in MOTIS, so there should be no problem with this syntax.

https://github.com/motis-project/nigiri/blob/master/src/loader/gtfs/parse_time.cc

But probably quoted time fields won't work (haven't tried it) - that's something we could improve if necessary.

[jbruechert]

In this case it's failing earlier on our side (gtfsclean), which we still need to filter other invalid stuff (wrong coordinates, too fast trips etc.)

[Altonss]

In this case it's failing earlier on our side (gtfsclean), which we still need to filter other invalid stuff (wrong coordinates, too fast trips etc.)

I think it is failing even earlier in https://github.com/public-transport/gtfsparser : all stop are badly formatted in HH:MM and therefor directly added to DroppedStopTimes I think, which leads to gtfsclean dropping them all.

[felixguendling]

HH:MM is something MOTIS should be able to work with - so maybe just disable fix?

[jbruechert]

Currently it still goes through gtfsclean anyway. We could add an option to skip that but then we loose filtering of fast trips for that feed which could break things.

I think making the gtfsparser more lenient will be the best solution.

[jbruechert]

Did this now, the next problem is

routes.txt:2 - Expected integer for field 'route_type', found 'TransporteAereo'

This feed must have seen zero testing or validation, I think it might be a case for an external data cleanup project.

[Altonss]

Did this now, the next problem is

routes.txt:2 - Expected integer for field 'route_type', found 'TransporteAereo'

This feed must have seen zero testing or validation, I think it might be a case for an external data cleanup project.

Wow this feed is amazingly bad 🤣 It would need a dedicated cleanup script at this point :/

[Altonss]

Did this now, the next problem is

routes.txt:2 - Expected integer for field 'route_type', found 'TransporteAereo'

This error is new, I have a version from january having the correct 1100 type :(

[jbruechert]

So after some more fiddling around, it is salvageable, but only with custom search-and-replace action and gtfsparser hacks.

In case someone wants to clean it up, I have good experiences with the free CI on gitlab.com. It allows to get a stable link to the latest artifact, so you can set up a cron job to do a number of sed commands there without much work, and add the output link here. Obviously anything else that allows to get a stable link works as well.

Manually cleaned up version

[felixguendling]

Maybe before putting in the effort to write a custom script, it would make sense to at least try to get in touch with the feed authors with a wish list which bugs would be most important to fix to have a usable feed. Maybe they are just not aware that their feed is effectively useless in the state they publish it.

[felixguendling]

I found it quite useful that the official validator produces a static report URL that I can put into the email with the note that this validator is the official one by the standardization organization itself, so it has some "authority".

https://gtfs-validator.mobilitydata.org/