State websites average ‘A’ grade for URL configuration, ‘F’ for sitemaps

State government websites averaged an “A” grade for the configurations of their URLs, but averaged an “F” when it comes to their sitemaps — the file that lists all of the pages on the website — according to a new analysis published this month.

The grades are based on new data shared on the “gov metadata” tracker, a new project from Luke Fretwell, a former journalist who now runs the tech firm ProudCity, and his son, Elias. The additional informational trackers are a continuation of a project from the father-son pair, who spent much of the past year scraping publicly available data of government websites, revealing last month that most fail to provide basic metadata for their users.

Just like metadata, both URLs — also known as web addresses — and sitemaps are important for accessibility and should be standardized to improve how the public interacts with government online, Fretwell told StateScoop via email.

“URLs are the first line of public engagement with the digital government experience. Proper URL configuration increases a sense of trust and ensures a smoother, more secure experience with government,” Frewell said. “Sitemaps make it easier for the public to access the information on a website, whether it’s via a search engine, AI bot or general research. It’s a public way of documenting all of the information located on a government website.”

The Fretwells evaluated URLs against three criteria: whether they use the secure HTTPS protocol, whether they can be accessed with and without ‘www’ included and if they use a sponsored top-level domain such as .gov, .edu or .mil.

They found 40 of the 50 states met all three criteria. (Delaware, Louisiana and Ohio, meanwhile, blocked their bots from scraping the data they needed for the analysis and weren’t included.)

“There’s no good excuse for having an improperly configured government URL,” Fretwell said. “Setting up HTTPS is relatively easy through organizations like Let’s Encrypt. Resolving www and non-www can be done via DNS management. And getting a top-level .gov domain is free through the Cybersecurity and Infrastructure Security Agency. There are some technical requirements getting each of these implemented, but overall it’s relatively easy, and the financial cost is minimal to mostly free.”

The Fretwells evaluated sitemaps against two criteria: that sitemap file was present and accessible, and that it was in XML file format. XML sitemaps include metadata for the website’s pages, such as when they were last updated, how often they usually change and how important they are compared to other pages on the site.

According to the analysis, 31 of 50 state websites were missing at least one criteria.

“By not having properly configured, standard sitemaps, government isn’t optimizing its content and services for search engines, AI bots or general research. This makes it more of a challenge for the public to find the information and services it needs,” Fretwell said. “Most content management systems automate sitemap creation and there are free tools that make it easy to implement these, so the technical and financial costs for this is also relatively low. There really isn’t an excuse for not having this.”

Fretwell said that while failing to meet web development standards makes the websites less accessible to the public, the matter should be treated more seriously during election season as disinformation campaigns become common.

“Failing to adhere to proper official URL standards adds to public confusion as to what’s official and what’s spoofed,” he said. “If a government URL has HTTPS and a .gov [top-level domain], the public can rest assured that their privacy is protected and they are truly visiting an official government website.”

Editor’s note: Luke Fretwell is a former employee of Scoop News Group.