Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add CloudQuery integration (ECOINT-11) #2451

Open
wants to merge 28 commits into
base: master
Choose a base branch
from

Conversation

erezrokah
Copy link

What does this PR do?

Add CloudQuery integration

Motivation

Get listed on the integrations page per https://docs.datadoghq.com/developers/integrations/?tab=integrations

Review checklist

  • PR has a meaningful title or PR has the no-changelog label attached
  • Feature or bugfix has tests
  • Git history is clean
  • If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo
  • If this PR includes a log pipeline, please add a description describing the remappers and processors.

Additional Notes

Anything else we should know when reviewing?

I followed the guide in https://docs.datadoghq.com/developers/integrations/?tab=integrations as best as I can. Feedback welcomed!

@erezrokah erezrokah requested review from a team as code owners July 29, 2024 10:06
@erezrokah erezrokah force-pushed the feat/cloudquery branch 2 times, most recently from adf28ee to f7469b0 Compare July 29, 2024 10:31
@drichards-87
Copy link
Contributor

Created Jira card for Docs Team review.

@drichards-87 drichards-87 added the editorial review Waiting on a more in-depth review from a docs team editor label Jul 29, 2024
Copy link
Contributor

@estherk15 estherk15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some suggestions for documentation, let me know if you have any questions or would like a re-review!

cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
erezrokah and others added 17 commits July 31, 2024 18:45
@erezrokah erezrokah requested a review from estherk15 July 31, 2024 15:56
@erezrokah
Copy link
Author

Thanks for the review @estherk15, great suggestions 🚀 I also followed up with 087f84a and 06a21ee to fix the links and lists numbering (as the lists numbering is not preserved after the code blocks).

@erezrokah
Copy link
Author

Thanks @estherk15, per a meeting with the team in charge of partner integrations, we'll submit this as a community integration until there's official OTEL support

@erezrokah erezrokah closed this Aug 5, 2024
@erezrokah erezrokah reopened this Aug 6, 2024
@erezrokah
Copy link
Author

erezrokah commented Aug 6, 2024

we'll submit this as a community integration until there's official OTEL support

OK so I guess I got confused by something, since the instructions in https://docs.datadoghq.com/agent/guide/use-community-integrations/?tab=agentv721v621 say to open a PR to this repo.

Can we accept this PR as a community integration?

Please note this integration does not require any installation on the agent side

@erezrokah erezrokah requested a review from a team as a code owner August 6, 2024 17:56
@erezrokah
Copy link
Author

Moving to draft until I fix the log pipelines CI

@erezrokah erezrokah marked this pull request as draft August 7, 2024 07:39
@erezrokah erezrokah marked this pull request as ready for review August 7, 2024 08:19
@erezrokah
Copy link
Author

Not sure how to fix these errors:
image

This is how I configured the filter in the Datadog UI for AWS:
image

We send a service field with name and version properties in the logs so we should be able to filter by those no?

kodiakhq bot pushed a commit to cloudquery/cloudquery that referenced this pull request Aug 7, 2024
#### Summary

This adds a new section to our Datadog dashboard with information on throttled APIs:
![image](https://github.com/user-attachments/assets/fda9fcfe-7d3c-4e42-a134-95890e64243a)

Clicking on a column will users open the logs in the relevant location:
![image](https://github.com/user-attachments/assets/e9b931b0-9074-467f-ad4a-b9ec4cd929f5)
![image](https://github.com/user-attachments/assets/06d2c975-df01-4a99-8446-317eb1b56c5d)

I've also added the the same change to DataDog/integrations-extras#2451.

DataDog/integrations-extras#2451 also adds custom log pipelines to extract the retry attempts into attributes instead of relying on the `retrying request` message.
However if we only distribute the dashboard (like we do on our website) we can't ship the pipelines with the dashboard (we need the integration for that).
@estherk15 estherk15 added docs/approved and removed editorial review Waiting on a more in-depth review from a docs team editor labels Aug 7, 2024
@erezrokah
Copy link
Author

erezrokah commented Aug 14, 2024

OK, removed the log pipelines in the initial version, I'll add those later

@erezrokah
Copy link
Author

14209ec marks this as a community integration

@mgashaj mgashaj changed the title feat: Add CloudQuery integration feat: Add CloudQuery integration (ECOINT-11) Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants