Only 100 access rules are generated.

[vitich]

Confirmation

• My issue isn't already found on the issue tracker.
• I have replicated my issue using the latest version of the library and it is still present.

cf-terraforming version

cf-terraforming -dev+

Expected outcome

I expected to migrate all access rules (>1000)

Actual outcome

Only 100 access rules are generated.

Steps to reproduce

Well... you should have > 100 access rules on CF

$cf-terraforming generate --resource-type cloudflare_access_rule  --zone XXX >> cloudflare.tf 2>/dev/null
terraform validate || exit 0
$ grep access_rule cloudflare.tf | wc -l
100

Tried several domain zones with more than 100 rules

References

No response

[Nathan-Yorio]

I would like to add I am experiencing the same issue just yesterday and found you beat me to the punch to post an issue 😄 . When I run cf-terraforming with verbose flag I get the following after the first 100 results page:

DEBU[0003] got unknown attribute configuration: key zone_id, value <nil>, value type <nil> 
DEBU[0003] got unknown attribute configuration: key notes, value <nil>, value type <nil> 

I thought this was interesting so I changed to zone ID instead and get

DEBU[0003] got unknown attribute configuration: key account_id, value <nil>, value type <nil>

I figure this is due to pagination and the api per_page limit is 100, but some nil values are breaking the switch case in the code when populating these attributes.

Also, I can retrieve all of them from the api just fine as long as I paginate the results, so it is a bug in cf-terraforming or cloudflare-go specifically it seems?

Edit:
I am not so familiar with the codebase or anything but perhaps this function is what is needed now because it supports params with pagination options? Assuming this is the same thing as an access rule?
https://github.com/cloudflare/cloudflare-go/blob/6f98ca9e9077ea6943b9cdc66f2dd7389cc58ade/ip_access_rules.go#L69

Edit2:
I also built the current ver from source and encountered the same issue when I run the binary

Edit3:
My bad I followed the trail and see that this issue is already intended to be offloaded to cloudflare-go. Any news of rough timeline for that?

[vitich]

This is a shame and disgrace for Cloudflare. For so many years, it has not been possible to make a normal integration with terraform - the problem of pagination is everywhere - dns, access rules, whatever. Specifically, I faced a problem - there are more than a thousand rules in my account (enterprise). Why can't they all be exported correctly? In fact, there is neither a convenient utility from Cloudflare nor an integration with terraform that would work correctly...