How to generate sitemap when using Airtable CMS (more than 20k chars)

Hi Divhunt team,

I am having an issue where i am creating a collection pages using Airtable as CMS Source.
With regards to generating a sitemap ideally we want to have an automatically generated sitemap from the airtable records however if I understand correctly it is not possible for now, the only potential solution i found is mentioned here

Usually when using 3rd-party database, you’ll need manually to add sitemap

The problem is there is a character limit of 20k characters, where the sitemap I want to upload is around 50k characters.

I also thought of using html sitemap but unfortunately that cant be used for GSC
since I imagine making the automatic sitemap from airtable is complicated, would it be possible to increase this sitemap character limit for now (to lets say 100k to be safe)?

Thanks in advance!

URL To Website : https://storefront.divhunt.art/

I’ve increased it to 100k, I think this value is big larger than we should allow, but we will monitor the performance and ram usage.

1 Like

Thanks for the quick response @dejan , however, im still receiving an errror.
is there something im missing?

Hi @dejan there still seems to be an error on uploading the sitemap, do you have any update regarding this ?

Can you share in pastebin sitemap you are trying to add? And website url? Thanks

sure, here is the sitemap i pasted to the input pastebin.com/raw/4tEtHnY1

and here is my site url https://rent.thestorefront.com

thanks @Pakic

Hi @Pakic @dejan any update on this ?

@Nicholas

Yeah, we won’t be able to increase char count, since it’s not an issue with length of sitemap, rather than allowing POST requests toward our servers, so increasing this will open up a security flaw in our system which can be exploited.

However there is a way to achieve this, using Workflows on “Sitemap” event, you can potentially fetch the external .txt file which holds your sitemap and show it instead the system one, in this way you can manage your sitemap elsewhere, just upload it anywhere, maybe you can upload it in Divhunt as .txt file as well.

So this seems to be the only way at the moment.

If something like this is fine with you, I can help you make it

hi @dejan ,
I see. yes, that can work for me. Let me know when the feature is live thanks!

Hi @dejan @Pakic , any update for this ?

Hi @dejan @Pakic , can I have an ETA for this sitemap feature? Thanks

@Nicholas

It’s ready, sorry for the wait.

Give me few hours to test it and I’ll integrate it into your website and let you know how it works.

Awesome, thankyou @dejan for the update ! :+1:

hi @dejan , were you able to deploy your change? do you have an ETA for this to be live ? thanks

@Nicholas

My deepest apologies for waiting, I was away, here’s a quick tutorial on how to achieve this.

Open workflows panel from top right corner (navbar).

  1. Create a new Workflow and name it Sitemap for example.
  2. Choose Event “Sitemap”
    image
  3. Add first action “GET CONTENTS”

    Provide URL and set variable name to sitemap.
  4. Add another action after “Echo” and set the value $sitemap
  5. And add the last action EXIT to prevent default sitemap loading.
  6. Publish workflows, publish website and test from incognito mode, since sitemap is not working when you’re logged in as admin.

Keep in mind that it’s not enough to just add URLs, you need to provide a full sitemap.xml structure in order to work.

Let me know if this works for you.

hi @dejan no worries.
Thankyou for the tutorial!

I am following it and got up to this point. However when I try to reach my sitemap, it appears to be showing an empty sitemap. Am i missing something?

Thankyou again for the assistance!

@Nicholas

Any chance you can add me as collaborator dejan@divhunt.com so I can check, since it’s working on my website.

hi @dejan access given. Lmk when you had a chance to look at it thankyou

@Nicholas

You haven’t published website, that’s why it didn’t work.

So it works now but your sitemap structure is invalid, you need to provide full sitemap xml structure to be loaded.

https://storefront.divhunt.art/sitemap.xml

1 Like

Ah, Shouldve seen that , apologies for the confusion. I fixed it and now its working.

Thankyou very much @dejan for your help on this!