diff --git a/README_ES.md b/README_ES.md
new file mode 100644
index 0000000000..d908eb4c6a
--- /dev/null
+++ b/README_ES.md
@@ -0,0 +1,369 @@
+
+
+ 
+
+
+Escáner de vulnerabilidades rápido y personalizable basado en un sencillo DSL basado en YAML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Cómo funciona •
+ Instalación •
+ Documentación •
+ Créditos •
+ Preguntas Frecuentes •
+ Únete a Discord
+
+
+
+ English •
+ 中文 •
+ Korean •
+ Indonesia •
+ Spanish
+
+
+---
+
+Nuclei se utiliza para enviar peticiones a múltiples objetivos basándose en una plantilla, lo que resulta en cero falsos positivos y proporciona un escaneo rápido en un gran número de hosts. Nuclei ofrece escaneos para una variedad de protocolos, incluyendo TCP, DNS, HTTP, SSL, File, Whois, Websocket, Headless, Code, etc. Con plantillas potentes y flexibles, Nuclei puede utilizarse para modelar todo tipo de comprobaciones de seguridad.
+
+Tenemos un [repositorio dedicado](https://github.com/projectdiscovery/nuclei-templates) que alberga varios tipos de plantillas de vulnerabilidades, contribuidas por **más de 300** investigadores y ingenieros de seguridad.
+
+## Cómo funciona
+
+
+
+ 
+
+
+
+| :exclamation: **Descargo de responsabilidad** |
+|---------------------------------|
+| **Este proyecto está en desarrollo activo**. Es de esperar que se produzcan cambios importantes con las nuevas versiones. Consulte el registro de cambios de la versión antes de actualizar. |
+| Este proyecto fue principalmente desarrollado para ser utilizado como una herramienta CLI independiente. **Ejecutar nuclei como un servicio puede suponer riesgos de seguridad.** Se recomienda utilizarlo con precaución y tomar medidas de seguridad adicionales. |
+
+# Instalación de Nuclei
+
+Nuclei requiere **go1.21** para instalarse correctamente. Ejecute el siguiente comando para instalar la última versión -
+
+```sh
+go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
+```
+
+
+ Brew
+
+ ```sh
+ brew install nuclei
+ ```
+
+
+
+ Docker
+
+ ```sh
+ docker pull projectdiscovery/nuclei:latest
+ ```
+
+
+
+**Más métodos de instalación [pueden encontrarse aquí](https://docs.projectdiscovery.io/tools/nuclei/install).**
+
+
+
+|
+
+### Plantillas de Nuclei
+
+Nuclei cuenta con soporte incorporado para la descarga/actualización automática de plantillas desde la versión [v2.5.2](https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2) en adelante. El proyecto [**Nuclei-Templates**](https://github.com/projectdiscovery/nuclei-templates) proporciona una lista de plantillas listas para usar, aportadas por la comunidad, y que se actualizan constantemente.
+
+También puedes utilizar la bandera `update-templates` para actualizar las plantillas de Nuclei en cualquier momento; puedes escribir tus propias pruebas para tu flujo de trabajo y necesidades individuales siguiendo la [guía de plantillas](https://docs.projectdiscovery.io/templates/) de Nuclei.
+
+La sintaxis de referencia YAML DSL está disponible [aquí](SYNTAX-REFERENCE.md).
+
+ |
+
+
+
+### Uso
+
+```sh
+nuclei -h
+```
+
+Esto mostrará ayuda sobre la herramienta. Aquí están todas las opciones que soporta.
+
+
+```console
+Nuclei es un escáner de vulnerabilidades rápido y basado en plantillas
+que se centra en su amplia configurabilidad, extensibilidad y facilidad de uso.
+
+Usage:
+ ./nuclei [flags]
+
+Flags:
+TARGET:
+ -u, -target string[] URLs/hosts a escanear
+ -l, -list string ruta al archivo que contiene la lista de URLs/hosts a escanear (uno por línea)
+ -eh, -exclude-hosts string[] hosts a excluir para escanear de la lista de entrada (ip, cidr, hostname)
+ -resume string reanudar el escaneo usando resume.cfg (la clusterización quedará inhabilitada)
+ -sa, -scan-all-ips escanear todas las IP asociadas al registro dns
+ -iv, -ip-version string[] versión IP a escanear del nombre de host (4,6) - (por defecto 4)
+
+TARGET-FORMAT:
+ -im, -input-mode string modo del archivo de entrada (list, burp, jsonl, yaml, openapi, swagger) (por defecto "list")
+ -ro, -required-only utilizar solo campos requeridos en el formato de entrada al generar peticiones
+ -sfv, -skip-format-validation saltar la validación de formato (como variables faltantes) al procesar el archivo de entrada
+
+TEMPLATES:
+ -nt, -new-templates ejecutar sólo las nuevas plantillas añadidas en la última versión de nuclei-templates
+ -ntv, -new-templates-version string[] ejecutar las nuevas plantillas añadidas en la versión especificada
+ -as, -automatic-scan escaneo web automático utilizando la detección de tecnología de wappalyzer para mapeo de etiquetas
+ -t, -templates string[] lista de plantillas o directorio de plantillas a ejecutar (separadas por comas, file)
+ -turl, -template-url string[] url de plantilla o lista que contiene urls de plantillas a ejecutar (separadas por comas, file)
+ -w, -workflows string[] lista de flujos de trabajo o directorio de flujos de trabajo a ejecutar (separadas por comas, file)
+ -wurl, -workflow-url string[] url de flujo de trabajo o lista que contiene urls de flujo de trabajo para ejecutar (separadas por comas, file)
+ -validate valida las plantillas pasadas a nuclei
+ -nss, -no-strict-syntax deshabilita la comprobación de sintaxis estricta en las plantillas
+ -td, -template-display muestra el contenido de las plantillas
+ -tl lista todas las plantillas disponibles
+ -tgl lista todas las etiquetas disponibles
+ -sign firma las plantillas con la clave privada definida en la variable de entorno NUCLEI_SIGNATURE_PRIVATE_KEY
+ -code habilita la carga de plantillas basadas en protocolos de código
+ -dut, -disable-unsigned-templates deshabilita la ejecución de plantillas no firmadas o plantillas con firma no coincidente
+
+FILTERING:
+ -a, -author string[] plantillas a ejecutar basadas en autores (separadas por comas, file)
+ -tags string[] plantillas a ejecutar basadas en etiquetas (separadas por comas, file)
+ -etags, -exclude-tags string[] plantillas a excluir basadas en etiquetas (separadas por comas, file)
+ -itags, -include-tags string[] etiquetas a ejecutar incluso si están excluidas ya sea por defecto o por configuración
+ -id, -template-id string[] plantillas a ejecutar basadas en IDs de plantilla (comma-separated, file, allow-wildcard)
+ -eid, -exclude-id string[] plantillas a excluir basadas en IDs de plantilla (separadas por comas, file)
+ -it, -include-templates string[] ruta al archivo de plantilla o directorio a ejecutar incluso si están excluidas ya sea por defecto o por configuración
+ -et, -exclude-templates string[] ruta al archivo de plantilla o directorio a excluir (separadas por comas, file)
+ -em, -exclude-matchers string[] matchers de plantilla a excluir en el resultado
+ -s, -severity value[] plantillas a ejecutar basadas en criticidad. Valores posibles: info, bajo, medio, alto, crítico, desconocido
+ -es, -exclude-severity value[] plantillas a excluir basadas en criticidad. Valores posibles: info, bajo, medio, alto, crítico, desconocido
+ -pt, -type value[] plantillas a ejecutar basadas en tipo de protocolo. Valores posibles: dns, file, http, headless, tcp, workflow, ssl, websocket, whois, code, javascript
+ -ept, -exclude-type value[] plantillas a excluir basadas en tipo de protocolo. Valores posibles: dns, file, http, headless, tcp, workflow, ssl, websocket, whois, code, javascript
+ -tc, -template-condition string[] plantillas a ejecutar basadas en condición de expresión
+
+OUTPUT:
+ -o, -output string archivo de salida donde guardar las incidencias/vulnerabilidades detectadas
+ -sresp, -store-resp almacenar todas las peticiones/respuestas enviadas por nuclei en el directorio de salida
+ -srd, -store-resp-dir string almacenar todas las peticiones/respuestas enviadas por nuclei en un directorio personalizado (por defecto "output")
+ -silent mostrar resultados únicamente
+ -nc, -no-color deshabilitar la coloración del contenido de salida (códigos de escape ANSI)
+ -j, -jsonl escribir la salida en formato JSONL(ines)
+ -irr, -include-rr -omit-raw incluir pares peticiones/respuesta en las salidas JSON, JSONL y Markdown (sólo para hallazgos) [OBSOLETO usar -omit-raw] (por defecto true)
+ -or, -omit-raw omitir los pares peticiones/respuesta en las salidas JSON, JSONL y Markdown (sólo para hallazgos)
+ -ot, -omit-template omitir plantilla codificada en la salida JSON, JSONL
+ -nm, -no-meta deshabilitar la impresión de metadatos de resultados en la salida cli
+ -ts, -timestamp habilitar la impresión de la marca de tiempo en la salida cli
+ -rdb, -report-db string base de datos de informes de nuclei (utilizarla siempre para persistir los datos de los informes)
+ -ms, -matcher-status mostrar el estado de fallo de coincidencia
+ -me, -markdown-export string directorio para exportar resultados en formato markdown
+ -se, -sarif-export string archivo para exportar resultados en formato SARIF
+ -je, -json-export string archivo para exportar resultados en formato JSON
+ -jle, -jsonl-export string archivo para exportar resultados en formato JSONL(ines)
+
+CONFIGURATIONS:
+ -config string ruta al archivo de configuración de nuclei
+ -fr, -follow-redirects habilitar el seguimiento de redirecciones para plantillas http
+ -fhr, -follow-host-redirects seguir redirecciones en el mismo host
+ -mr, -max-redirects int número máximo de redirecciones a seguir para plantillas http (por defecto 10)
+ -dr, -disable-redirects deshabilitar redirecciones para plantillas http
+ -rc, -report-config string archivo de configuración del módulo de informes de nuclei
+ -H, -header string[] encabezado/cookie personalizado a incluir en todas las peticiones http en formato header:value (cli, file)
+ -V, -var value variables personalizadas en formato key=value
+ -r, -resolvers string archivo que contiene lista de resolutores para nuclei
+ -sr, -system-resolvers utilizar resolución de DNS del sistema como fallback de error
+ -dc, -disable-clustering deshabilitar la clusterización de peticiones
+ -passive habilitar el modo de procesamiento pasivo de respuestas HTTP
+ -fh2, -force-http2 forzar la conexión http2 en las peticiones
+ -ev, -env-vars habilitar el uso de variables de entorno en la plantilla
+ -cc, -client-cert string archivo de certificado de cliente (codificado en PEM) utilizado para autenticarse contra los hosts escaneados
+ -ck, -client-key string archivo de clave de cliente (codificado en PEM) utilizado para autenticarse contra los hosts escaneados
+ -ca, -client-ca string archivo de autoridad de certificación de cliente (codificado en PEM) utilizado para autenticarse contra los hosts escaneados
+ -sml, -show-match-line mostrar líneas de coincidencia para plantillas de archivo, funciona solo con extractores
+ -ztls utilizar la biblioteca ztls con autofallback a estándar para tls13 [Obsoleto] autofallback a ztls está habilitado por defecto
+ -sni string nombre de host tls sni a usar (por defecto: nombre de dominio de entrada)
+ -dt, -dialer-timeout value tiempo de espera para peticiones de red
+ -dka, -dialer-keep-alive value duración de keep-alive para peticiones de red
+ -lfa, -allow-local-file-access permite el acceso a archivos (carga útil) en cualquier lugar del sistema
+ -lna, -restrict-local-network-access bloquea conexiones a la red local / privada
+ -i, -interface string interfaz de red a usar para el escaneo de red
+ -at, -attack-type string tipo de combinaciones de carga útil a realizar (batteringram, pitchfork, clusterbomb)
+ -sip, -source-ip string dirección ip de origen a usar para el escaneo de red
+ -rsr, -response-size-read int tamaño máximo de respuesta a leer en bytes (por defecto 10485760)
+ -rss, -response-size-save int tamaño máximo de respuesta a guardar en bytes (por defecto 1048576)
+ -reset reset elimina todos los archivos de configuración y datos de nuclei (incluidas las nuclei-templates)
+ -tlsi, -tls-impersonate habilitar client hello (ja3) tls randomization experimental
+
+INTERACTSH:
+ -iserver, -interactsh-server string url del servidor interactsh para instancia autoalojada (por defecto: oast.pro,oast.live,oast.site,oast.online,oast.fun,oast.me)
+ -itoken, -interactsh-token string token de autenticación del servidor interactsh autoalojado
+ -interactions-cache-size int número de peticiones a mantener en la caché de interacciones (por defecto 5000)
+ -interactions-eviction int número de segundos a esperar antes de eliminar las solicitudes de la caché (por defecto 60)
+ -interactions-poll-duration int número de segundos a esperar antes de cada solicitud de polling de interacciones (por defecto 5)
+ -interactions-cooldown-period int tiempo adicional para el polling de interacciones antes de salir (por defecto 5)
+ -ni, -no-interactsh desactivar el servidor interactsh para pruebas OAST, excluir plantillas basadas en OAST
+
+FUZZING:
+ -ft, -fuzzing-type string sobrescribe el tipo de fuzzing establecido en la plantilla (replace, prefix, postfix, infix)
+ -fm, -fuzzing-mode string sobrescribe el modo de fuzzing establecido en la plantilla (multiple, single)
+ -fuzz habilita la carga de plantillas de fuzzing (Obsoleto: usar -dast en su lugar)
+ -dast solo ejecuta plantillas DAST
+
+UNCOVER:
+ -uc, -uncover habilita el motor uncover
+ -uq, -uncover-query string[] consulta de búsqueda uncover
+ -ue, -uncover-engine string[] motor de búsqueda uncover (shodan,censys,fofa,shodan-idb,quake,hunter,zoomeye,netlas,criminalip,publicwww,hunterhow) (por defecto shodan)
+ -uf, -uncover-field string campos uncover a devolver (ip,port,host) (por defecto "ip:port")
+ -ul, -uncover-limit int resultados uncover a devolver (por defecto 100)
+ -ur, -uncover-ratelimit int sobrescribe el límite de velocidad de los motores con el límite de velocidad del motor uncover (por defecto 60 req/min) (por defecto 60)
+
+RATE-LIMIT:
+ -rl, -rate-limit int número máximo de peticiones a enviar por segundo (por defecto 150)
+ -rlm, -rate-limit-minute int número máximo de peticiones a enviar por minuto
+ -bs, -bulk-size int número máximo de hosts a ser analizados en paralelo por plantilla (por defecto 25)
+ -c, -concurrency int número máximo de plantillas a ejecutar en paralelo (por defecto 25)
+ -hbs, -headless-bulk-size int número máximo de hosts headless a ser analizados en paralelo por plantilla (por defecto 10)
+ -headc, -headless-concurrency int número máximo de plantillas headless a ejecutar en paralelo (por defecto 10)
+ -jsc, -js-concurrency int número máximo de entornos de ejecución de JavaScript a ejecutar en paralelo (por defecto 120)
+ -pc, -payload-concurrency int concurrencia máxima de carga útil para cada plantilla (por defecto 25)
+
+OPTIMIZATIONS:
+ -timeout int tiempo de espera en segundos (por defecto 10)
+ -retries int número de veces que se reintenta una petición fallida (por defecto 1)
+ -ldp, -leave-default-ports dejar puertos HTTP/HTTPS predeterminados (por ejemplo, host:80,host:443)
+ -mhe, -max-host-error int errores máximos para un host antes de omitirlo del escaneo (por defecto 30)
+ -te, -track-error string[] agrega el error dado a la lista de seguimiento de errores máximos por host (standard, file)
+ -nmhe, -no-mhe deshabilita la omisión del host del escaneo basado en errores
+ -project utiliza una carpeta de proyecto para evitar enviar la misma petición varias veces
+ -project-path string establece una ruta de proyecto específica (por defecto "/tmp")
+ -spm, -stop-at-first-match detiene el procesamiento de las peticiones HTTP después de la primera coincidencia (puede romper la lógica de la plantilla/flujo de trabajo)
+ -stream modo transmisión - comienza a trabajar sin ordenar la entrada
+ -ss, -scan-strategy value estrategia a utilizar mientras se escanea (auto/host-spray/template-spray) (por defecto auto)
+ -irt, -input-read-timeout value tiempo de espera en la lectura de entrada (por defecto 3m0s)
+ -nh, -no-httpx deshabilita análisis httpx para entradas que no son URL
+ -no-stdin deshabilita el procesamiento de la entrada estándar
+
+HEADLESS:
+ -headless habilita las plantillas que requieren soporte de navegadores sin interfaz gráfica (headless browser) (el usuario root en Linux deshabilitará el sandbox)
+ -page-timeout int segundos para esperar cada página en modo sin interfaz (por defecto 20)
+ -sb, -show-browser muestra el navegador en la pantalla al ejecutar plantillas con modo sin interfaz
+ -ho, -headless-options string[] inicia Chrome en modo sin interfaz con opciones adicionales
+ -sc, -system-chrome utiliza el navegador Chrome instalado localmente en lugar del instalado por nuclei
+ -lha, -list-headless-action lista de acciones sin interfaz disponibles
+
+DEBUG:
+ -debug muestra todas las peticiones y respuestas
+ -dreq, -debug-req muestra todas las peticiones enviadas
+ -dresp, -debug-resp muestra todas las respuestas recibidas
+ -p, -proxy string[] lista de proxies http/socks5 a utilizar (separados por comas o archivo de entrada)
+ -pi, -proxy-internal proxy para todas las peticiones internas
+ -ldf, -list-dsl-function lista todas las firmas de función DSL admitidas
+ -tlog, -trace-log string archivo a escribir el registro de traza de peticiones enviadas
+ -elog, -error-log string archivo a escribir el registro de error de peticiones enviadas
+ -version muestra la versión de nuclei
+ -hm, -hang-monitor habilita la monitorización de bloqueos de nuclei
+ -v, -verbose muestra salida detallada
+ -profile-mem string archivo opcional de volcado de memoria de nuclei
+ -vv muestra las plantillas cargadas para el escaneo
+ -svd, -show-var-dump muestra el volcado de variables para depuración
+ -ep, -enable-pprof habilita el servidor de depuración pprof
+ -tv, -templates-version muestra la versión de las plantillas nuclei (nuclei-templates) instaladas
+ -hc, -health-check ejecuta comprobación de diagnóstico
+
+UPDATE:
+ -up, -update actualiza el motor de nuclei a la última versión lanzada
+ -ut, -update-templates actualiza nuclei-templates a la última versión lanzada
+ -ud, -update-template-dir string directorio personalizado para instalar/actualizar nuclei-templates
+ -duc, -disable-update-check deshabilita la comprobación automática de actualizaciones de nuclei/templates
+
+STATISTICS:
+ -stats muestra estadísticas sobre el escaneo en ejecución
+ -sj, -stats-json muestra estadísticas en formato JSONL(ines)
+ -si, -stats-interval int número de segundos a esperar entre mostrar una actualización de estadísticas (por defecto 5)
+ -mp, -metrics-port int puerto para exponer métricas de nuclei (por defecto 9092)
+
+CLOUD:
+ -auth configura la clave de API del cloud de projectdiscovery (pdcp)
+ -cup, -cloud-upload sube los resultados del escaneo al dashboard de pdcp
+ -sid, -scan-id string sube los resultados del escaneo al ID de escaneo dado
+
+AUTHENTICATION:
+ -sf, -secret-file string[] ruta al archivo de configuración que contiene los secrets para el escaneo autenticado de nuclei
+ -ps, -prefetch-secrets precarga los secrets del archivo de secrets
+
+
+EXAMPLES:
+Ejecutar nuclei en un solo host:
+ $ nuclei -target example.com
+
+Ejecutar nuclei con directorios de plantillas específicos:
+ $ nuclei -target example.com -t http/cves/ -t ssl
+
+Ejecutar nuclei contra una lista de hosts:
+ $ nuclei -list hosts.txt
+
+Ejecutar nuclei con una salida JSON:
+ $ nuclei -target example.com -json-export output.json
+
+Ejecutar nuclei con salidas Markdown ordenadas (con variables de entorno):
+ $ MARKDOWN_EXPORT_SORT_MODE=template nuclei -target example.com -markdown-export nuclei_report/
+
+Documentación adicional disponible en: https://docs.nuclei.sh/getting-started/running
+```
+
+### Ejecutando Nuclei
+
+Consulta https://docs.projectdiscovery.io/tools/nuclei/running para obtener detalles sobre cómo ejecutar Nuclei.
+
+### Uso de Nuclei desde código Go
+
+La guía completa sobre cómo usar Nuclei como biblioteca/SDK está disponible en [godoc](https://pkg.go.dev/github.com/projectdiscovery/nuclei/v3/lib#section-readme).
+
+
+### Recursos
+
+Puedes acceder a la documentación principal de Nuclei en https://docs.projectdiscovery.io/tools/nuclei/, y obtener más información sobre Nuclei en la nube con [ProjectDiscovery Cloud Platform](https://cloud.projectdiscovery.io).
+
+¡Consulta https://docs.projectdiscovery.io/tools/nuclei/resources para obtener más recursos y videos sobre Nuclei!
+
+### Créditos
+
+Gracias a todos los increíbles [contribuyentes de la comunidad que enviaron PRs](https://github.com/projectdiscovery/nuclei/graphs/contributors) y mantienen este proyecto actualizado. :heart:
+
+Si tienes una idea o algún tipo de mejora, eres bienvenido a contribuir y participar en el Proyecto, siéntete libre de enviar tu PR.
+
+
+
+ 
+
+
+
+
+También echa un vistazo a los siguientes proyectos de código abierto similares que pueden adaptarse a tu flujo de trabajo:
+
+[FFuF](https://github.com/ffuf/ffuf), [Qsfuzz](https://github.com/ameenmaali/qsfuzz), [Inception](https://github.com/proabiral/inception), [Snallygaster](https://github.com/hannob/snallygaster), [Gofingerprint](https://github.com/Static-Flow/gofingerprint), [Sn1per](https://github.com/1N3/Sn1per/tree/master/templates), [Google tsunami](https://github.com/google/tsunami-security-scanner), [Jaeles](https://github.com/jaeles-project/jaeles), [ChopChop](https://github.com/michelin/ChopChop)
+
+### Licencia
+
+Nuclei se distribuye bajo la [Licencia MIT](https://github.com/projectdiscovery/nuclei/blob/main/LICENSE.md)
+
+
+ 
+
diff --git a/README_ID.md b/README_ID.md
index cb679c059f..702ef3f23b 100644
--- a/README_ID.md
+++ b/README_ID.md
@@ -32,7 +32,8 @@
English •
中文 •
Korean •
- Indonesia
+ Indonesia •
+ Spanish
---
@@ -55,7 +56,15 @@ Kami memiliki [repositori khusus](https://github.com/projectdiscovery/nuclei-tem
Nuclei membutuhkan **go1.21** agar dapat diinstall. Jalankan perintah berikut untuk menginstal versi terbaru -
```sh
+<<<<<<< HEAD
go install -v github.com/Explorer1092/nuclei/v3/cmd/nuclei@latest
+=======
+<<<<<<< HEAD
+go install -v github.com/Explorer1092/nuclei/v2/cmd/nuclei@latest
+=======
+go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a
+>>>>>>> projectdiscovery-main
```
**Metode [instalasi lain dapat ditemukan di sini](https://nuclei.projectdiscovery.io/nuclei/get-started/).**
diff --git a/README_JP.md b/README_JP.md
new file mode 100644
index 0000000000..f1988e3569
--- /dev/null
+++ b/README_JP.md
@@ -0,0 +1,172 @@
+
+
+
+
+
+シンプルなYAMLベースのDSLに基づいた高速でカスタマイズ可能な脆弱性スキャナー
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 動作原理 •
+ インストール •
+ ドキュメント •
+ クレジット •
+ FAQ •
+ Discordに参加
+
+
+
+ 英語 •
+ 中国語 •
+ 韓国語 •
+ インドネシア語 •
+ スペイン語
+
+
+---
+
+Nucleiは、テンプレートに基づいてターゲット間でリクエストを送信するために使用され、偽陽性がゼロであり、多数のホストで高速なスキャンを提供します。Nucleiは、TCP、DNS、HTTP、SSL、ファイル、Whois、Websocket、Headless、Codeなど、さまざまなプロトコルのスキャンを提供します。強力で柔軟なテンプレートを使用して、Nucleiはすべての種類のセキュリティチェックをモデル化するために使用できます。
+
+**300人以上の** セキュリティ研究者およびエンジニアが提供するさまざまなタイプの脆弱性テンプレートを収容する[専用リポジトリ](https://github.com/projectdiscovery/nuclei-templates)を持っています。
+
+## 動作原理
+
+
+
+
+
+| :exclamation: **免責事項** |
+|---------------------------------|
+| **このプロジェクトは積極的に開発されています**。リリースによって重大な変更が発生することがあります。更新する前にリリースの変更ログを確認してください。 |
+| このプロジェクトは主にスタンドアロンのCLIツールとして使用されることを目的として構築されました。**Nucleiをサービスとして実行すると、セキュリティリスクが生じる可能性があります。**注意して使用し、追加のセキュリティ対策を講じることをお勧めします。 |
+
+# Nucleiのインストール
+
+Nucleiを正常にインストールするには、**go1.21**が必要です。以下のコマンドを実行して最新バージョンをインストールしてください -
+
+```sh
+go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
+```
+
+
+ Brew
+
+ ```sh
+ brew install nuclei
+ ```
+
+
+
+ Docker
+
+ ```sh
+ docker pull projectdiscovery/nuclei:latest
+ ```
+
+
+
+**より多くのインストール方法は[こちら](https://docs.projectdiscovery.io/tools/nuclei/install)で見つけることができます。**
+
+
+
+|
+
+### Nucleiテンプレート
+
+Nucleiは、バージョン[v2.5.2](https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2)以降、デフォルトでテンプレートの自動ダウンロード/更新をサポートしています。[**Nuclei-Templates**](https://github.com/projectdiscovery/nuclei-templates)プロジェクトは、常に更新されるコミュニティ提供の即時使用可能なテンプレートのリストを提供します。
+
+`update-templates`フラグを使用して、いつでもNucleiテンプレートを更新することができます。Nucleiの[テンプレートガイド](https://docs.projectdiscovery.io/templates/)に従って、個々のワークフローとニーズに合わせた独自のチェックを作成することができます。
+
+YAML DSLの構文リファレンスは[こちら](SYNTAX-REFERENCE.md)で確認できます。
+
+ |
+
+
+
+### 使用方法
+
+```sh
+nuclei -h
+```
+
+これにより、ツールのヘルプが表示されます。ここには、サポートされているすべてのスイッチがあります。
+
+```console
+Nucleiは、広範な設定可能性、大規模な拡張性、および使いやすさに焦点を当てた、
+高速でテンプレートベースの脆弱性スキャナーです。
+
+使用法:
+ ./nuclei [flags]
+
+フラグ:
+ターゲット:
+ -u, -target string[] スキャンする対象のURL/ホスト
+ -l, -list string スキャンする対象のURL/ホストのリストが含まれているファイルへのパス（1行に1つ）
+ -resume string resume.cfgを使用してスキャンを再開（クラスタリングは無効になります）
+ -sa, -scan-all-ips DNSレコードに関連付けられているすべてのIPをスキャン
+ -iv, -ip-version string[] ホスト名のスキャンするIPバージョン（4,6）-（デフォルトは4）
+
+テンプレート:
+ -nt, -new-templates 最新のnuclei-templatesリリースに追加された新しいテンプレートのみを実行
+ -ntv, -new-templates-version string[] 特定のバージョンに追加された新しいテンプレートを実行
+ -as, -automatic-scan wappalyzer技術検出をタグマッピングに使用した自動Webスキャン
+ -t, -templates string[] 実行するテンプレートまたはテンプレートディレクトリのリスト（カンマ区切り、ファイル）
+ -turl, -template-url string[] 実行するテンプレートのURLまたはテンプレートURLのリスト（カンマ区切り、ファイル）
+ -w, -workflows string[] 実行するワークフローまたはワークフローディレクトリのリスト（カンマ区切り、ファイル）
+ -wurl, -workflow-url string[] 実行するワークフローのURLまたはワークフローURLのリスト（カンマ区切り、ファイル）
+ -validate Nucleiに渡されたテンプレートを検証
+ -nss, -no-strict-syntax テンプレートで厳密な構文チェックを無効にする
+ -td, -template-display テンプレートの内容を表示
+ -tl 利用可能なすべてのテンプレートをリスト
+ -sign NUCLEI_SIGNATURE_PRIVATE_KEY環境変数で定義された秘密鍵でテンプレートに署名
+ -code コードプロトコルベースのテンプレートのロードを有効にする
+
+フィルタリング:
+ -a, -author string[] 作者に基づいて実行するテンプレート（カンマ区切り、ファイル）
+ -tags string[] タグに基づいて実行するテンプレート（カンマ区切り、ファイル）
+ -etags, -exclude-tags string[] タグに基づいて除外するテンプレート（カンマ区切り、ファイル）
+ -itags, -include-tags string[] デフォルトまたは設定によって除外されている場合でも実行する必要があるタグ
+ -id, -template-id string[] テンプレートIDに基づいて実行するテンプレート（カンマ区切り、ファイル）
+ -eid, -exclude-id string[] テンプレートIDに基づいて除外するテンプレート（カンマ区切り、ファイル）
+ -it, -include-templates string[] デフォルトまたは設定によって除外されている場合でも実行する必要があるテンプレート
+ -et, -exclude-templates string[] 除外するテンプレートまたはテンプレートディレクトリへのパス（カンマ区切り、ファイル）
+ -em, -exclude-matchers string[] 結果で除外するテンプレートマッチャー
+ -s, -severity value[] 重大度に基づいて実行するテンプレート。可能な値：info, low, medium, high, critical, unknown
+ -es, -exclude-severity value[] 重大度に基づいて除外するテンプレート。可能な値：info, low, medium, high, critical, unknown
+ -pt, -type value[] プロトコルタイプに基づいて実行するテンプレート。可能な値：dns, file, http, headless, tcp, workflow, ssl, websocket, whois, code, javascript
+ -ept, -exclude-type value[] プロトコルタイプに基づいて除外するテンプレート。可能な値：dns, file, http, headless, tcp, workflow, ssl, websocket, whois, code, javascript
+ -tc, -template-condition string[] 式条件に基づいて実行するテンプレート
+
+出力:
+ -o, -output string 発見された問題/脆弱性を書き込む出力ファイル
+ -sresp, -store-resp Nucleiを通じて渡されたすべてのリクエスト/レスポンスを出力ディレクトリに保存
+ -srd, -store-resp-dir string Nucleiを通じて渡されたすべてのリクエスト/レスポンスをカスタムディレクトリに保存（デフォルトは「output」）
+ -silent 結果のみを表示
+ -nc, -no-color 出力内容の着色を無効にする（ANSIエスケープコード）
+ -j, -jsonl JSONL(ines)形式で出力を書き込む
+ -irr, -include-rr -omit-raw JSON、JSONL、Markdown出力にリクエスト/レスポンスペアを含める（発見のみ）[非推奨 -omit-raw使用]（デフォルトはtrue）
+ -or, -omit-raw JSON、JSONL、Markdown出力でリクエスト/レスポンスペアを省略する（発見のみ）
+ -ot, -omit-template JSON、JSONL出力でエンコードされたテンプレートを省略
+ -nm, -no-meta CLI出力で結果のメタデータの印刷を無効にする
+ -ts, -timestamp CLI出力にタイムスタンプを印刷することを有効にする
+ -rdb, -report-db string Nucleiレポートデータベース（レポートデータを永続化するために常にこれを使用）
+ -ms, -matcher-status マッチ失敗のステータスを表示
+ -me, -markdown-export string Markdown形式で結果をエクスポートするディレクトリ
+ -se, -sarif-export string SARIF形式で結果をエクスポートするファイル
+ -je, -json-export string JSON形式で結果をエクスポートするファイル
+ -jle, -jsonl-export string JSONL(ine)形式で結果をエクスポートするファイル
+
+設定:
+ -config string Nucleiの設定ファイルへのパス
+ -fr, -follow-redirects HTTPテンプレートのリダイレクトをフォローすることを有効にする
+ -fhr, -follow-host-redirects
\ No newline at end of file
diff --git a/README_KR.md b/README_KR.md
index 7cd262b094..72c002e90f 100644
--- a/README_KR.md
+++ b/README_KR.md
@@ -30,7 +30,8 @@
English •
中文 •
- 한국어
+ 한국어 •
+ 스페인어
---
@@ -53,7 +54,15 @@ Nuclei는 템플릿을 기반으로 대상 간에 요청을 보내기 위해 사
Nuclei를 성공적으로 설치하기 위해서 **go1.21**가 필요합니다. 다음 명령을 실행하여 최신 버전을 설치합니다.
```sh
+<<<<<<< HEAD
go install -v github.com/Explorer1092/nuclei/v3/cmd/nuclei@latest
+=======
+<<<<<<< HEAD
+go install -v github.com/Explorer1092/nuclei/v2/cmd/nuclei@latest
+=======
+go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a
+>>>>>>> projectdiscovery-main
```
**자세한 설치 방법은 [여기](https://nuclei.projectdiscovery.io/nuclei/get-started/)에서 찾을 수 있습니다.**
diff --git a/SYNTAX-REFERENCE.md b/SYNTAX-REFERENCE.md
index 7fa62c5f44..24cbd04d38 100755
--- a/SYNTAX-REFERENCE.md
+++ b/SYNTAX-REFERENCE.md
@@ -1373,6 +1373,19 @@ Fuzzing describes schema to fuzz http requests
+
+
+Analyzer is an analyzer to use for matching the response.
+
+
+
+
+
+
+
self-contained bool
@@ -1404,6 +1417,19 @@ Valid values:
+skip-secret-file bool
+
+
+
+
+SkipSecretFile skips the authentication or authorization configured in the secret file.
+
+
+
+
+
+
+
cookie-reuse bool
@@ -1638,6 +1664,19 @@ FuzzPreConditionOperator is the operator between multiple PreConditions for fuzz
+
+
+global-matchers bool
+
+
+
+
+GlobalMatchers marks matchers as static and applies globally to all result events from other templates
+
+
+
+
+
@@ -1800,13 +1839,50 @@ Valid values:
Part is the part of request to fuzz.
-query fuzzes the query part of url. More parts will be added later.
+
+Valid values:
+
+
+ - query
+
+ - header
+
+ - path
+
+ - body
+
+ - cookie
+
+ - request
+
+
+
+
+
+
+parts []string
+
+
+
+
+Parts is the list of parts to fuzz. If multiple parts need to be
+defined while excluding some, this should be used instead of singular part.
Valid values:
- query
+
+ - header
+
+ - path
+
+ - body
+
+ - cookie
+
+ - request
@@ -1962,6 +2038,59 @@ Appears in:
+## analyzers.AnalyzerTemplate
+AnalyzerTemplate is the template for the analyzer
+
+Appears in:
+
+
+- http.Request.analyzer
+
+
+
+
+
+
+
+
+
+name string
+
+
+
+
+Name is the name of the analyzer to use
+
+
+Valid values:
+
+
+ - time_delay
+
+
+
+
+
+
+parameters map[string]interface{}
+
+
+
+
+Parameters is the parameters for the analyzer
+
+Parameters are different for each analyzer. For example, you can customize
+time_delay analyzer with sleep_duration, time_slope_error_range, etc. Refer
+to the docs for each analyzer to get an idea about parameters.
+
+
+
+
+
+
+
+
+
## SignatureTypeHolder
SignatureTypeHolder is used to hold internal type of the signature
@@ -3160,6 +3289,19 @@ read-all: false
+
+
+stop-at-first-match bool
+
+
+
+
+StopAtFirstMatch stops the execution of the requests and template as soon as a match is found.
+
+
+
+
+
@@ -3614,8 +3756,18 @@ Enum Values:
- files
+ - waitdom
+
+ - waitfcp
+
+ - waitfmp
+
+ - waitidle
+
- waitload
+ - waitstable
+
- getresource
- extract
@@ -3632,6 +3784,8 @@ Enum Values:
- waitevent
+ - dialog
+
- keyboard
- debug
@@ -3702,11 +3856,33 @@ Appears in:
Part Definitions:
+- template-id - ID of the template executed
+- template-info - Info Block of the template executed
+- template-path - Path of the template executed
+- host - Host is the input to the template
+- port - Port is the port of the host
+- matched - Matched is the input which was matched upon
- type - Type is the type of request made
+- timestamp - Timestamp is the time when the request was made
- response - JSON SSL protocol handshake details
+- cipher - Cipher is the encryption algorithm used
+- domains - Domains are the list of domain names in the certificate
+- fingerprint_hash - Fingerprint hash is the unique identifier of the certificate
+- ip - IP is the IP address of the server
+- issuer_cn - Issuer CN is the common name of the certificate issuer
+- issuer_dn - Issuer DN is the distinguished name of the certificate issuer
+- issuer_org - Issuer organization is the organization of the certificate issuer
- not_after - Timestamp after which the remote cert expires
-- host - Host is the input to the template
-- matched - Matched is the input which was matched upon
+- not_before - Timestamp before which the certificate is not valid
+- probe_status - Probe status indicates if the probe was successful
+- serial - Serial is the serial number of the certificate
+- sni - SNI is the server name indication used in the handshake
+- subject_an - Subject AN is the list of subject alternative names
+- subject_cn - Subject CN is the common name of the certificate subject
+- subject_dn - Subject DN is the distinguished name of the certificate subject
+- subject_org - Subject organization is the organization of the certificate subject
+- tls_connection - TLS connection is the type of TLS connection used
+- tls_version - TLS version is the version of the TLS protocol used
@@ -4302,19 +4478,6 @@ Code contains code to execute for the javascript request.
-timeout int
-
-
-
-
-Timeout in seconds is optional timeout for each javascript script execution (i.e init, pre-condition, code)
-
-
-
-
-
-
-
stop-at-first-match bool
diff --git a/cmd/docgen/docgen.go b/cmd/docgen/docgen.go
index 685bb3f0f8..8d1486ce31 100644
--- a/cmd/docgen/docgen.go
+++ b/cmd/docgen/docgen.go
@@ -11,7 +11,15 @@ import (
"github.com/invopop/jsonschema"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/templates"
+=======
+<<<<<<< HEAD:v2/cmd/docgen/docgen.go
+ "github.com/Explorer1092/nuclei/v2/pkg/templates"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/docgen/docgen.go
+>>>>>>> projectdiscovery-main
)
var pathRegex = regexp.MustCompile(`github\.com/Explorer1092/nuclei/v3/(?:internal|pkg)/(?:.*/)?([A-Za-z.]+)`)
diff --git a/cmd/functional-test/main.go b/cmd/functional-test/main.go
index a79b8ee519..362364543b 100644
--- a/cmd/functional-test/main.go
+++ b/cmd/functional-test/main.go
@@ -8,16 +8,24 @@ import (
"os"
"strings"
+ "github.com/kitabisa/go-ci"
"github.com/logrusorgru/aurora"
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/functional-test/main.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/functional-test/main.go
+>>>>>>> projectdiscovery-main
)
var (
- success = aurora.Green("[✓]").String()
- failed = aurora.Red("[✘]").String()
- githubAction = os.Getenv("GH_ACTION") == "true"
+ success = aurora.Green("[✓]").String()
+ failed = aurora.Red("[✘]").String()
mainNucleiBinary = flag.String("main", "", "Main Branch Nuclei Binary")
devNucleiBinary = flag.String("dev", "", "Dev Branch Nuclei Binary")
@@ -45,7 +53,7 @@ func runFunctionalTests(debug bool) (error, bool) {
errored, failedTestCases := runTestCases(file, debug)
- if githubAction {
+ if ci.IsCI() {
fmt.Println("::group::Failed tests with debug")
for _, failedTestCase := range failedTestCases {
_ = runTestCase(failedTestCase, true)
diff --git a/cmd/functional-test/run.sh b/cmd/functional-test/run.sh
index 15c61ff23a..189b1936de 100755
--- a/cmd/functional-test/run.sh
+++ b/cmd/functional-test/run.sh
@@ -20,7 +20,15 @@ echo "::group::Installing nuclei templates"
echo "::endgroup::"
echo "::group::Building latest release of nuclei"
+<<<<<<< HEAD
go build -o nuclei$extension -v github.com/Explorer1092/nuclei/v3/cmd/nuclei
+=======
+<<<<<<< HEAD:v2/cmd/functional-test/run.sh
+go build -o nuclei$extension -v github.com/Explorer1092/nuclei/v2/cmd/nuclei
+=======
+go build -o nuclei$extension -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/functional-test/run.sh
+>>>>>>> projectdiscovery-main
echo "::endgroup::"
echo 'Starting Nuclei functional test'
diff --git a/cmd/integration-test/code.go b/cmd/integration-test/code.go
index c89d26904c..0a7db84e43 100644
--- a/cmd/integration-test/code.go
+++ b/cmd/integration-test/code.go
@@ -99,7 +99,7 @@ type codePreCondition struct{}
// Execute executes a test case and returns an error if occurred
func (h *codePreCondition) Execute(filePath string) error {
- results, err := testutils.RunNucleiArgsWithEnvAndGetResults(debug, getEnvValues(), "-t", filePath, "-u", "input", "-code")
+ results, err := testutils.RunNucleiArgsWithEnvAndGetResults(debug, getEnvValues(), "-t", filePath, "-u", "input", "-code", "-esc")
if err != nil {
return err
}
diff --git a/cmd/integration-test/custom-dir.go b/cmd/integration-test/custom-dir.go
index c87cc411ca..70a47efb9f 100644
--- a/cmd/integration-test/custom-dir.go
+++ b/cmd/integration-test/custom-dir.go
@@ -3,7 +3,15 @@ package main
import (
"os"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/custom-dir.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/custom-dir.go
+>>>>>>> projectdiscovery-main
)
type customConfigDirTest struct{}
diff --git a/cmd/integration-test/dns.go b/cmd/integration-test/dns.go
index 9e50d0c1af..a84961ed43 100644
--- a/cmd/integration-test/dns.go
+++ b/cmd/integration-test/dns.go
@@ -1,7 +1,15 @@
package main
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/dns.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/dns.go
+>>>>>>> projectdiscovery-main
)
var dnsTestCases = []TestCaseInfo{
diff --git a/cmd/integration-test/file.go b/cmd/integration-test/file.go
index 757d0e27bc..c0e2483bed 100644
--- a/cmd/integration-test/file.go
+++ b/cmd/integration-test/file.go
@@ -1,7 +1,15 @@
package main
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/file.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/file.go
+>>>>>>> projectdiscovery-main
)
var fileTestcases = []TestCaseInfo{
@@ -15,7 +23,7 @@ type fileWithOrMatcher struct{}
// Execute executes a test case and returns an error if occurred
func (h *fileWithOrMatcher) Execute(filePath string) error {
- results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "protocols/file/data/", debug)
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "protocols/file/data/", debug, "-file")
if err != nil {
return err
}
@@ -27,7 +35,7 @@ type fileWithAndMatcher struct{}
// Execute executes a test case and returns an error if occurred
func (h *fileWithAndMatcher) Execute(filePath string) error {
- results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "protocols/file/data/", debug)
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "protocols/file/data/", debug, "-file")
if err != nil {
return err
}
@@ -39,7 +47,7 @@ type fileWithExtractor struct{}
// Execute executes a test case and returns an error if occurred
func (h *fileWithExtractor) Execute(filePath string) error {
- results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "protocols/file/data/", debug)
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "protocols/file/data/", debug, "-file")
if err != nil {
return err
}
diff --git a/cmd/integration-test/flow.go b/cmd/integration-test/flow.go
index 273e045bf5..1ff13f02cb 100644
--- a/cmd/integration-test/flow.go
+++ b/cmd/integration-test/flow.go
@@ -22,7 +22,7 @@ var flowTestcases = []TestCaseInfo{
type conditionalFlow struct{}
func (t *conditionalFlow) Execute(filePath string) error {
- results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "blog.projectdiscovery.io", debug)
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "cloud.projectdiscovery.io", debug)
if err != nil {
return err
}
diff --git a/cmd/integration-test/fuzz.go b/cmd/integration-test/fuzz.go
index ca1634b7b4..125290ace3 100644
--- a/cmd/integration-test/fuzz.go
+++ b/cmd/integration-test/fuzz.go
@@ -10,6 +10,16 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
"github.com/julienschmidt/httprouter"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/fuzz.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/fuzz.go
+>>>>>>> projectdiscovery-main
)
const (
diff --git a/cmd/integration-test/headless.go b/cmd/integration-test/headless.go
index 37280413ce..3cb791d805 100644
--- a/cmd/integration-test/headless.go
+++ b/cmd/integration-test/headless.go
@@ -7,7 +7,15 @@ import (
"github.com/julienschmidt/httprouter"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/headless.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/headless.go
+>>>>>>> projectdiscovery-main
)
var headlessTestcases = []TestCaseInfo{
diff --git a/cmd/integration-test/http.go b/cmd/integration-test/http.go
index 4335e8aa18..46be764940 100644
--- a/cmd/integration-test/http.go
+++ b/cmd/integration-test/http.go
@@ -17,12 +17,21 @@ import (
"github.com/julienschmidt/httprouter"
"gopkg.in/yaml.v2"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/http.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/http.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
errorutil "github.com/projectdiscovery/utils/errors"
logutil "github.com/projectdiscovery/utils/log"
sliceutil "github.com/projectdiscovery/utils/slice"
stringsutil "github.com/projectdiscovery/utils/strings"
+ unitutils "github.com/projectdiscovery/utils/unit"
)
var httpTestcases = []TestCaseInfo{
@@ -154,7 +163,7 @@ func (h *httpInteractshRequest) Execute(filePath string) error {
return err
}
- return expectResultsCount(results, 1)
+ return expectResultsCount(results, 1, 2)
}
type httpDefaultMatcherCondition struct{}
@@ -509,7 +518,7 @@ func (h *httpPostMultipartBody) Execute(filePath string) error {
var routerErr error
router.POST("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
- if err := r.ParseMultipartForm(1 * 1024); err != nil {
+ if err := r.ParseMultipartForm(unitutils.Mega); err != nil {
routerErr = err
return
}
@@ -951,7 +960,7 @@ func (h *httpRequestSelfContained) Execute(filePath string) error {
}()
defer server.Close()
- results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug)
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug, "-esc")
if err != nil {
return err
}
@@ -987,7 +996,7 @@ func (h *httpRequestSelfContainedWithParams) Execute(filePath string) error {
}()
defer server.Close()
- results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug)
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug, "-esc")
if err != nil {
return err
}
@@ -1030,7 +1039,7 @@ func (h *httpRequestSelfContainedFileInput) Execute(filePath string) error {
}
defer FileLoc.Close()
- results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug, "-V", "test="+FileLoc.Name())
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug, "-V", "test="+FileLoc.Name(), "-esc")
if err != nil {
return err
}
diff --git a/cmd/integration-test/integration-test.go b/cmd/integration-test/integration-test.go
index 15031823e4..144e73dceb 100644
--- a/cmd/integration-test/integration-test.go
+++ b/cmd/integration-test/integration-test.go
@@ -7,11 +7,22 @@ import (
"runtime"
"strings"
+ "github.com/kitabisa/go-ci"
"github.com/logrusorgru/aurora"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
"github.com/Explorer1092/nuclei/v3/pkg/testutils/fuzzplayground"
"github.com/projectdiscovery/gologger"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/integration-test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/gologger"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils/fuzzplayground"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/integration-test.go
+>>>>>>> projectdiscovery-main
sliceutil "github.com/projectdiscovery/utils/slice"
)
@@ -22,10 +33,9 @@ type TestCaseInfo struct {
}
var (
- debug = os.Getenv("DEBUG") == "true"
- githubAction = os.Getenv("GH_ACTION") == "true"
- customTests = os.Getenv("TESTS")
- protocol = os.Getenv("PROTO")
+ debug = os.Getenv("DEBUG") == "true"
+ customTests = os.Getenv("TESTS")
+ protocol = os.Getenv("PROTO")
success = aurora.Green("[✓]").String()
failed = aurora.Red("[✘]").String()
@@ -55,6 +65,7 @@ var (
"dsl": dslTestcases,
"flow": flowTestcases,
"javascript": jsTestcases,
+ "matcher-status": matcherStatusTestcases,
}
// flakyTests are run with a retry count of 3
flakyTests = map[string]bool{
@@ -102,11 +113,19 @@ func main() {
failedTestTemplatePaths := runTests(customTestsList)
if len(failedTestTemplatePaths) > 0 {
- if githubAction {
- debug = true
- fmt.Println("::group::Failed integration tests in debug mode")
- _ = runTests(failedTestTemplatePaths)
+ if ci.IsCI() {
+ // run failed tests again assuming they are flaky
+ // if they fail as well only then we assume that there is an actual issue
+ fmt.Println("::group::Running failed tests again")
+ failedTestTemplatePaths = runTests(failedTestTemplatePaths)
fmt.Println("::endgroup::")
+
+ if len(failedTestTemplatePaths) > 0 {
+ debug = true
+ fmt.Println("::group::Failed integration tests in debug mode")
+ _ = runTests(failedTestTemplatePaths)
+ fmt.Println("::endgroup::")
+ }
}
os.Exit(1)
diff --git a/cmd/integration-test/library.go b/cmd/integration-test/library.go
index 29bc6a9eda..212a76c4e9 100644
--- a/cmd/integration-test/library.go
+++ b/cmd/integration-test/library.go
@@ -31,6 +31,44 @@ import (
"github.com/logrusorgru/aurora"
"github.com/pkg/errors"
"github.com/projectdiscovery/goflags"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/code.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/disk"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/loader"
+ "github.com/Explorer1092/nuclei/v2/pkg/core"
+ "github.com/Explorer1092/nuclei/v2/pkg/core/inputs"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/parsers"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/hosterrorscache"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/interactsh"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolinit"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolstate"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
+ "github.com/projectdiscovery/nuclei/v3/pkg/core"
+ "github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
+ parsers "github.com/projectdiscovery/nuclei/v3/pkg/loader/workflow"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hosterrorscache"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/library.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/ratelimit"
)
diff --git a/cmd/integration-test/loader.go b/cmd/integration-test/loader.go
index 5a4a519cc9..aed9af3909 100644
--- a/cmd/integration-test/loader.go
+++ b/cmd/integration-test/loader.go
@@ -9,9 +9,19 @@ import (
"github.com/julienschmidt/httprouter"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
errorutil "github.com/projectdiscovery/utils/errors"
permissionutil "github.com/projectdiscovery/utils/permission"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/loader.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+ errorutil "github.com/projectdiscovery/utils/errors"
+ permissionutil "github.com/projectdiscovery/utils/permission"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/loader.go
+>>>>>>> projectdiscovery-main
)
var loaderTestcases = []TestCaseInfo{
@@ -172,7 +182,14 @@ func (h *nonExistentTemplateList) Execute(nonExistingTemplateList string) error
ts := httptest.NewServer(router)
defer ts.Close()
- _, err := testutils.RunNucleiBareArgsAndGetResults(debug, nil, "-target", ts.URL, "-template-url", ts.URL+"/404")
+ configFileData := `remote-template-domain: [ "` + ts.Listener.Addr().String() + `" ]`
+ err := os.WriteFile("test-config.yaml", []byte(configFileData), permissionutil.ConfigFilePermission)
+ if err != nil {
+ return err
+ }
+ defer os.Remove("test-config.yaml")
+
+ _, err = testutils.RunNucleiBareArgsAndGetResults(debug, nil, "-target", ts.URL, "-template-url", ts.URL+"/404", "-config", "test-config.yaml")
if err == nil {
return fmt.Errorf("expected error for nonexisting workflow url")
}
@@ -188,7 +205,14 @@ func (h *nonExistentWorkflowList) Execute(nonExistingWorkflowList string) error
ts := httptest.NewServer(router)
defer ts.Close()
- _, err := testutils.RunNucleiBareArgsAndGetResults(debug, nil, "-target", ts.URL, "-workflow-url", ts.URL+"/404")
+ configFileData := `remote-template-domain: [ "` + ts.Listener.Addr().String() + `" ]`
+ err := os.WriteFile("test-config.yaml", []byte(configFileData), permissionutil.ConfigFilePermission)
+ if err != nil {
+ return err
+ }
+ defer os.Remove("test-config.yaml")
+
+ _, err = testutils.RunNucleiBareArgsAndGetResults(debug, nil, "-target", ts.URL, "-workflow-url", ts.URL+"/404", "-config", "test-config.yaml")
if err == nil {
return fmt.Errorf("expected error for nonexisting workflow url")
}
diff --git a/cmd/integration-test/matcher-status.go b/cmd/integration-test/matcher-status.go
new file mode 100644
index 0000000000..d89f7c46b0
--- /dev/null
+++ b/cmd/integration-test/matcher-status.go
@@ -0,0 +1,120 @@
+package main
+
+import (
+ "encoding/json"
+ "fmt"
+ "strings"
+
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+)
+
+var matcherStatusTestcases = []TestCaseInfo{
+ {Path: "protocols/http/get.yaml", TestCase: &httpNoAccess{}},
+ {Path: "protocols/network/net-https.yaml", TestCase: &networkNoAccess{}},
+ {Path: "protocols/headless/headless-basic.yaml", TestCase: &headlessNoAccess{}},
+ {Path: "protocols/javascript/net-https.yaml", TestCase: &javascriptNoAccess{}},
+ {Path: "protocols/websocket/basic.yaml", TestCase: &websocketNoAccess{}},
+ {Path: "protocols/dns/a.yaml", TestCase: &dnsNoAccess{}},
+}
+
+type httpNoAccess struct{}
+
+func (h *httpNoAccess) Execute(filePath string) error {
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "trust_me_bro.real", debug, "-ms", "-j")
+ if err != nil {
+ return err
+ }
+ event := &output.ResultEvent{}
+ _ = json.Unmarshal([]byte(results[0]), event)
+ expectedError := "no address found for host"
+ if !strings.Contains(event.Error, expectedError) {
+ return fmt.Errorf("unexpected result: expecting \"%s\" error but got \"%s\"", expectedError, event.Error)
+ }
+ return nil
+}
+
+type networkNoAccess struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *networkNoAccess) Execute(filePath string) error {
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "trust_me_bro.real", debug, "-ms", "-j")
+ if err != nil {
+ return err
+ }
+ event := &output.ResultEvent{}
+ _ = json.Unmarshal([]byte(results[0]), event)
+
+ if event.Error != "no address found for host" {
+ return fmt.Errorf("unexpected result: expecting \"no address found for host\" error but got \"%s\"", event.Error)
+ }
+ return nil
+}
+
+type headlessNoAccess struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *headlessNoAccess) Execute(filePath string) error {
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "trust_me_bro.real", debug, "-headless", "-ms", "-j")
+ if err != nil {
+ return err
+ }
+ event := &output.ResultEvent{}
+ _ = json.Unmarshal([]byte(results[0]), event)
+
+ if event.Error == "" {
+ return fmt.Errorf("unexpected result: expecting an error but got \"%s\"", event.Error)
+ }
+ return nil
+}
+
+type javascriptNoAccess struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *javascriptNoAccess) Execute(filePath string) error {
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "trust_me_bro.real", debug, "-ms", "-j")
+ if err != nil {
+ return err
+ }
+ event := &output.ResultEvent{}
+ _ = json.Unmarshal([]byte(results[0]), event)
+
+ if event.Error == "" {
+ return fmt.Errorf("unexpected result: expecting an error but got \"%s\"", event.Error)
+ }
+ return nil
+}
+
+type websocketNoAccess struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *websocketNoAccess) Execute(filePath string) error {
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "ws://trust_me_bro.real", debug, "-ms", "-j")
+ if err != nil {
+ return err
+ }
+ event := &output.ResultEvent{}
+ _ = json.Unmarshal([]byte(results[0]), event)
+
+ if event.Error == "" {
+ return fmt.Errorf("unexpected result: expecting an error but got \"%s\"", event.Error)
+ }
+ return nil
+}
+
+type dnsNoAccess struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *dnsNoAccess) Execute(filePath string) error {
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "trust_me_bro.real", debug, "-ms", "-j")
+ if err != nil {
+ return err
+ }
+ event := &output.ResultEvent{}
+ _ = json.Unmarshal([]byte(results[0]), event)
+
+ if event.Error == "" {
+ return fmt.Errorf("unexpected result: expecting an error but got \"%s\"", event.Error)
+ }
+ return nil
+}
diff --git a/cmd/integration-test/multi.go b/cmd/integration-test/multi.go
index 7d970bd2ef..cc6e13592c 100644
--- a/cmd/integration-test/multi.go
+++ b/cmd/integration-test/multi.go
@@ -14,7 +14,7 @@ type multiProtoDynamicExtractor struct{}
// Execute executes a test case and returns an error if occurred
func (h *multiProtoDynamicExtractor) Execute(templatePath string) error {
- results, err := testutils.RunNucleiTemplateAndGetResults(templatePath, "blog.projectdiscovery.io", debug)
+ results, err := testutils.RunNucleiTemplateAndGetResults(templatePath, "docs.projectdiscovery.io", debug)
if err != nil {
return err
}
diff --git a/cmd/integration-test/network.go b/cmd/integration-test/network.go
index ffbb658462..ec20e9cd88 100644
--- a/cmd/integration-test/network.go
+++ b/cmd/integration-test/network.go
@@ -119,7 +119,7 @@ func (h *networkRequestSelContained) Execute(filePath string) error {
_, _ = conn.Write([]byte("Authentication successful"))
})
defer ts.Close()
- results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug)
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, "", debug, "-esc")
if err != nil {
return err
}
diff --git a/cmd/integration-test/offline-http.go b/cmd/integration-test/offline-http.go
index 8c44e5ce47..2714d0ebf9 100644
--- a/cmd/integration-test/offline-http.go
+++ b/cmd/integration-test/offline-http.go
@@ -3,7 +3,15 @@ package main
import (
"fmt"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/offline-http.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/offline-http.go
+>>>>>>> projectdiscovery-main
)
var offlineHttpTestcases = []TestCaseInfo{
diff --git a/cmd/integration-test/ssl.go b/cmd/integration-test/ssl.go
index 34cf07caa8..3cb4022370 100644
--- a/cmd/integration-test/ssl.go
+++ b/cmd/integration-test/ssl.go
@@ -4,7 +4,15 @@ import (
"crypto/tls"
"net"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/ssl.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/ssl.go
+>>>>>>> projectdiscovery-main
)
var sslTestcases = []TestCaseInfo{
@@ -13,6 +21,7 @@ var sslTestcases = []TestCaseInfo{
{Path: "protocols/ssl/custom-cipher.yaml", TestCase: &sslCustomCipher{}},
{Path: "protocols/ssl/custom-version.yaml", TestCase: &sslCustomVersion{}},
{Path: "protocols/ssl/ssl-with-vars.yaml", TestCase: &sslWithVars{}},
+ {Path: "protocols/ssl/multi-req.yaml", TestCase: &sslMultiReq{}},
}
type sslBasic struct{}
@@ -118,3 +127,23 @@ func (h *sslWithVars) Execute(filePath string) error {
return expectResultsCount(results, 1)
}
+
+type sslMultiReq struct{}
+
+func (h *sslMultiReq) Execute(filePath string) error {
+ ts := testutils.NewTCPServer(&tls.Config{}, defaultStaticPort, func(conn net.Conn) {
+ defer conn.Close()
+ data := make([]byte, 4)
+ if _, err := conn.Read(data); err != nil {
+ return
+ }
+ })
+ defer ts.Close()
+
+ results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-V")
+ if err != nil {
+ return err
+ }
+
+ return expectResultsCount(results, 2)
+}
diff --git a/cmd/integration-test/template-dir.go b/cmd/integration-test/template-dir.go
index 0a263f2983..cf60b69601 100644
--- a/cmd/integration-test/template-dir.go
+++ b/cmd/integration-test/template-dir.go
@@ -3,7 +3,15 @@ package main
import (
"os"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/template-dir.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/template-dir.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
)
diff --git a/cmd/integration-test/template-path.go b/cmd/integration-test/template-path.go
index 30e8c9797d..b55ced12e8 100644
--- a/cmd/integration-test/template-path.go
+++ b/cmd/integration-test/template-path.go
@@ -4,8 +4,18 @@ import (
"fmt"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/template-path.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/template-path.go
+>>>>>>> projectdiscovery-main
)
func getTemplatePath() string {
diff --git a/cmd/integration-test/websocket.go b/cmd/integration-test/websocket.go
index 57a598d23c..73320e9f6a 100644
--- a/cmd/integration-test/websocket.go
+++ b/cmd/integration-test/websocket.go
@@ -6,7 +6,15 @@ import (
"github.com/gobwas/ws/wsutil"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/websocket.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/websocket.go
+>>>>>>> projectdiscovery-main
)
var websocketTestCases = []TestCaseInfo{
diff --git a/cmd/integration-test/whois.go b/cmd/integration-test/whois.go
index f20e168460..327262634f 100644
--- a/cmd/integration-test/whois.go
+++ b/cmd/integration-test/whois.go
@@ -1,7 +1,15 @@
package main
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/cmd/integration-test/whois.go
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/integration-test/whois.go
+>>>>>>> projectdiscovery-main
)
var whoisTestCases = []TestCaseInfo{
diff --git a/cmd/integration-test/workflow.go b/cmd/integration-test/workflow.go
index ee387ade3b..a6cf6c0f45 100644
--- a/cmd/integration-test/workflow.go
+++ b/cmd/integration-test/workflow.go
@@ -10,9 +10,16 @@ import (
"github.com/julienschmidt/httprouter"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/templates"
"github.com/Explorer1092/nuclei/v3/pkg/templates/signer"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/signer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+ sliceutil "github.com/projectdiscovery/utils/slice"
+>>>>>>> projectdiscovery-main
)
var workflowTestcases = []TestCaseInfo{
@@ -25,6 +32,7 @@ var workflowTestcases = []TestCaseInfo{
{Path: "workflow/dns-value-share-workflow.yaml", TestCase: &workflowDnsKeyValueShare{}},
{Path: "workflow/code-value-share-workflow.yaml", TestCase: &workflowCodeKeyValueShare{}, DisableOn: isCodeDisabled}, // isCodeDisabled declared in code.go
{Path: "workflow/multiprotocol-value-share-workflow.yaml", TestCase: &workflowMultiProtocolKeyValueShare{}},
+ {Path: "workflow/multimatch-value-share-workflow.yaml", TestCase: &workflowMultiMatchKeyValueShare{}},
{Path: "workflow/shared-cookie.yaml", TestCase: &workflowSharedCookies{}},
}
@@ -229,6 +237,44 @@ func (h *workflowMultiProtocolKeyValueShare) Execute(filePath string) error {
return expectResultsCount(results, 2)
}
+type workflowMultiMatchKeyValueShare struct{}
+
+// Execute executes a test case and returns an error if occurred
+func (h *workflowMultiMatchKeyValueShare) Execute(filePath string) error {
+ var receivedData []string
+ router := httprouter.New()
+ router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+ fmt.Fprintf(w, "This is test matcher text")
+ })
+ router.GET("/path1", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+ fmt.Fprintf(w, "href=\"test-value-%s\"", r.URL.Query().Get("v"))
+ })
+ router.GET("/path2", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
+ body, _ := io.ReadAll(r.Body)
+ receivedData = append(receivedData, string(body))
+ fmt.Fprintf(w, "test-value")
+ })
+ ts := httptest.NewServer(router)
+ defer ts.Close()
+
+ results, err := testutils.RunNucleiWorkflowAndGetResults(filePath, ts.URL, debug)
+ if err != nil {
+ return err
+ }
+
+ // Check if we received the data from both request to /path1 and it is not overwritten by the later one.
+ // They will appear in brackets because of another bug: https://github.com/orgs/projectdiscovery/discussions/3766
+ if !sliceutil.Contains(receivedData, "[test-value-1]") || !sliceutil.Contains(receivedData, "[test-value-2]") {
+ return fmt.Errorf(
+ "incorrect data: did not receive both extracted data from the first request!\nReceived Data:\n\t%s\nResults:\n\t%s",
+ strings.Join(receivedData, "\n\t"),
+ strings.Join(results, "\n\t"),
+ )
+ }
+ // The number of expected results is 3: the workflow's Matcher Name based condition check forwards both match, and the other branch with simple subtemplates goes with one
+ return expectResultsCount(results, 3)
+}
+
type workflowSharedCookies struct{}
// Execute executes a test case and returns an error if occurred
diff --git a/cmd/nuclei/issue-tracker-config.yaml b/cmd/nuclei/issue-tracker-config.yaml
index 51778eb0ad..07ccf828f2 100644
--- a/cmd/nuclei/issue-tracker-config.yaml
+++ b/cmd/nuclei/issue-tracker-config.yaml
@@ -142,4 +142,34 @@
# # Username for the elasticsearch instance
# username: test
# # Password is the password for elasticsearch instance
-# password: test
\ No newline at end of file
+# password: test
+#linear:
+# # api-key is the API key for the linear account
+# api-key: ""
+# # allow-list sets a tracker level filter to only create issues for templates with
+# # these severity labels or tags (does not affect exporters. set those globally)
+# deny-list:
+# severity: critical
+# # deny-list sets a tracker level filter to never create issues for templates with
+# # these severity labels or tags (does not affect exporters. set those globally)
+# deny-list:
+# severity: low
+# # team-id is the ID of the team in Linear
+# team-id: ""
+# # project-id is the ID of the project in Linear
+# project-id: ""
+# # duplicate-issue-check flag to enable duplicate tracking issue check
+# duplicate-issue-check: false
+# # open-state-id is the ID of the open state in Linear
+# open-state-id: ""
+#mongodb:
+# # the connection string to the MongoDB database
+# # (e.g., mongodb://root:example@localhost:27017/nuclei?ssl=false&authSource=admin)
+# connection-string: ""
+# # the name of the collection to store the issues
+# collection-name: ""
+# # excludes the Request and Response from the results (helps with filesize)
+# omit-raw: false
+# # determines the number of results to be kept in memory before writing it to the database or 0 to
+# # persist all in memory and write all results at the end (default)
+# batch-size: 0
\ No newline at end of file
diff --git a/cmd/nuclei/main.go b/cmd/nuclei/main.go
index f20376f064..18ef67f8a1 100644
--- a/cmd/nuclei/main.go
+++ b/cmd/nuclei/main.go
@@ -9,9 +9,11 @@ import (
"path/filepath"
"runtime"
"runtime/pprof"
+ "runtime/trace"
"strings"
"time"
+ _pdcp "github.com/projectdiscovery/nuclei/v3/internal/pdcp"
"github.com/projectdiscovery/utils/auth/pdcp"
"github.com/projectdiscovery/utils/env"
_ "github.com/projectdiscovery/utils/pprof"
@@ -36,8 +38,40 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/gologger/levels"
"github.com/projectdiscovery/interactsh/pkg/client"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/cmd/nuclei/main.go
+ "github.com/Explorer1092/nuclei/v2/internal/runner"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/common/dsl"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/uncover"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http"
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+ "github.com/Explorer1092/nuclei/v2/pkg/types/scanstrategy"
+ "github.com/Explorer1092/nuclei/v2/pkg/utils/monitor"
+=======
+ "github.com/projectdiscovery/nuclei/v3/internal/runner"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
+ "github.com/projectdiscovery/nuclei/v3/pkg/installer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/uncover"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/extensions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/signer"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types/scanstrategy"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/monitor"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:cmd/nuclei/main.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
+ unitutils "github.com/projectdiscovery/utils/unit"
updateutils "github.com/projectdiscovery/utils/update"
)
@@ -101,26 +135,52 @@ func main() {
return
}
- // Profiling related code
+ // Profiling & tracing related code
if memProfile != "" {
- f, err := os.Create(memProfile)
+ memProfile = strings.TrimSuffix(memProfile, filepath.Ext(memProfile)) + ".prof"
+ memProfileFile, err := os.Create(memProfile)
if err != nil {
- gologger.Fatal().Msgf("profile: could not create memory profile %q: %v", memProfile, err)
+ gologger.Fatal().Msgf("profile: could not create memory profile %q file: %v", memProfile, err)
}
- old := runtime.MemProfileRate
+
+ traceFilepath := strings.TrimSuffix(memProfile, filepath.Ext(memProfile)) + ".trace"
+ traceFile, err := os.Create(traceFilepath)
+ if err != nil {
+ gologger.Fatal().Msgf("profile: could not create trace %q file: %v", traceFilepath, err)
+ }
+
+ oldMemProfileRate := runtime.MemProfileRate
runtime.MemProfileRate = 4096
- gologger.Print().Msgf("profile: memory profiling enabled (rate %d), %s", runtime.MemProfileRate, memProfile)
+
+ // Start tracing
+ if err := trace.Start(traceFile); err != nil {
+ gologger.Fatal().Msgf("profile: could not start trace: %v", err)
+ }
defer func() {
- _ = pprof.Lookup("heap").WriteTo(f, 0)
- f.Close()
- runtime.MemProfileRate = old
- gologger.Print().Msgf("profile: memory profiling disabled, %s", memProfile)
+ // Start CPU profiling
+ if err := pprof.WriteHeapProfile(memProfileFile); err != nil {
+ gologger.Fatal().Msgf("profile: could not start CPU profile: %v", err)
+ }
+ memProfileFile.Close()
+ traceFile.Close()
+ trace.Stop()
+ runtime.MemProfileRate = oldMemProfileRate
+
+ gologger.Info().Msgf("Memory profile saved at %q", memProfile)
+ gologger.Info().Msgf("Traced at %q", traceFilepath)
}()
}
runner.ParseOptions(options)
+ if options.ScanUploadFile != "" {
+ if err := runner.UploadResultsToCloud(options); err != nil {
+ gologger.Fatal().Msgf("could not upload scan results to cloud dashboard: %s\n", err)
+ }
+ return
+ }
+
nucleiRunner, err := runner.New(options)
if err != nil {
gologger.Fatal().Msgf("Could not create runner: %s\n", err)
@@ -141,7 +201,10 @@ func main() {
nucleiRunner.Close()
gologger.Info().Msgf("Creating resume file: %s\n", resumeFileName)
err := nucleiRunner.SaveResumeConfig(resumeFileName)
- return errorutil.NewWithErr(err).Msgf("couldn't create crash resume file")
+ if err != nil {
+ return errorutil.NewWithErr(err).Msgf("couldn't create crash resume file")
+ }
+ return nil
})
}
@@ -231,6 +294,8 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.BoolVar(&options.SignTemplates, "sign", false, "signs the templates with the private key defined in NUCLEI_SIGNATURE_PRIVATE_KEY env variable"),
flagSet.BoolVar(&options.EnableCodeTemplates, "code", false, "enable loading code protocol-based templates"),
flagSet.BoolVarP(&options.DisableUnsignedTemplates, "disable-unsigned-templates", "dut", false, "disable running unsigned templates or templates with mismatched signature"),
+ flagSet.BoolVarP(&options.EnableSelfContainedTemplates, "enable-self-contained", "esc", false, "enable loading self-contained templates"),
+ flagSet.BoolVar(&options.EnableFileTemplates, "file", false, "enable loading file templates"),
)
flagSet.CreateGroup("filters", "Filtering",
@@ -268,6 +333,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.StringVarP(&options.SarifExport, "sarif-export", "se", "", "file to export results in SARIF format"),
flagSet.StringVarP(&options.JSONExport, "json-export", "je", "", "file to export results in JSON format"),
flagSet.StringVarP(&options.JSONLExport, "jsonl-export", "jle", "", "file to export results in JSONL(ine) format"),
+ flagSet.StringSliceVarP(&options.Redact, "redact", "rd", nil, "redact given list of keys from query parameter, request header and body", goflags.CommaSeparatedStringSliceOptions),
)
flagSet.CreateGroup("configs", "Configurations",
@@ -293,7 +359,6 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.BoolVarP(&options.ShowMatchLine, "show-match-line", "sml", false, "show match lines for file templates, works with extractors only"),
flagSet.BoolVar(&options.ZTLS, "ztls", false, "use ztls library with autofallback to standard one for tls13 [Deprecated] autofallback to ztls is enabled by default"), //nolint:all
flagSet.StringVar(&options.SNI, "sni", "", "tls sni hostname to use (default: input domain name)"),
- flagSet.DurationVarP(&options.DialerTimeout, "dialer-timeout", "dt", 0, "timeout for network requests."),
flagSet.DurationVarP(&options.DialerKeepAlive, "dialer-keep-alive", "dka", 0, "keep-alive duration for network requests."),
flagSet.BoolVarP(&options.AllowLocalFileAccess, "allow-local-file-access", "lfa", false, "allows file (payload) access anywhere on the system"),
flagSet.BoolVarP(&options.RestrictLocalNetworkAccess, "restrict-local-network-access", "lna", false, "blocks connections to the local / private network"),
@@ -301,8 +366,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.StringVarP(&options.AttackType, "attack-type", "at", "", "type of payload combinations to perform (batteringram,pitchfork,clusterbomb)"),
flagSet.StringVarP(&options.SourceIP, "source-ip", "sip", "", "source ip address to use for network scan"),
flagSet.IntVarP(&options.ResponseReadSize, "response-size-read", "rsr", 0, "max response size to read in bytes"),
- flagSet.IntVarP(&options.ResponseSaveSize, "response-size-save", "rss", 1*1024*1024, "max response size to read in bytes"),
- flagSet.DurationVarP(&options.ResponseReadTimeout, "response-read-timeout", "rrt", time.Duration(5*time.Second), "response read timeout in seconds"),
+ flagSet.IntVarP(&options.ResponseSaveSize, "response-size-save", "rss", unitutils.Mega, "max response size to read in bytes"),
flagSet.CallbackVar(resetCallback, "reset", "reset removes all nuclei configuration and data files (including nuclei-templates)"),
flagSet.BoolVarP(&options.TlsImpersonate, "tls-impersonate", "tlsi", false, "enable experimental client hello (ja3) tls randomization"),
flagSet.StringVarP(&options.HttpApiEndpoint, "http-api-endpoint", "hae", "", "experimental http api endpoint"),
@@ -323,6 +387,9 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.StringVarP(&options.FuzzingMode, "fuzzing-mode", "fm", "", "overrides fuzzing mode set in template (multiple, single)"),
flagSet.BoolVar(&fuzzFlag, "fuzz", false, "enable loading fuzzing templates (Deprecated: use -dast instead)"),
flagSet.BoolVar(&options.DAST, "dast", false, "enable / run dast (fuzz) nuclei templates"),
+ flagSet.BoolVarP(&options.DisplayFuzzPoints, "display-fuzz-points", "dfp", false, "display fuzz points in the output for debugging"),
+ flagSet.IntVar(&options.FuzzParamFrequency, "fuzz-param-frequency", 10, "frequency of uninteresting parameters for fuzzing before skipping"),
+ flagSet.StringVarP(&options.FuzzAggressionLevel, "fuzz-aggression", "fa", "low", "fuzzing aggression level controls payload count for fuzz (low, medium, high)"),
)
flagSet.CreateGroup("uncover", "Uncover",
@@ -388,9 +455,10 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.CallbackVar(printVersion, "version", "show nuclei version"),
flagSet.BoolVarP(&options.HangMonitor, "hang-monitor", "hm", false, "enable nuclei hang monitoring"),
flagSet.BoolVarP(&options.Verbose, "verbose", "v", false, "show verbose output"),
- flagSet.StringVar(&memProfile, "profile-mem", "", "optional nuclei memory profile dump file"),
+ flagSet.StringVar(&memProfile, "profile-mem", "", "generate memory (heap) profile & trace files"),
flagSet.BoolVar(&options.VerboseVerbose, "vv", false, "display templates loaded for scan"),
flagSet.BoolVarP(&options.ShowVarDump, "show-var-dump", "svd", false, "show variables dump for debugging"),
+ flagSet.IntVarP(&options.VarDumpLimit, "var-dump-limit", "vdl", 255, "limit the number of characters displayed in var dump"),
flagSet.BoolVarP(&options.EnablePprof, "enable-pprof", "ep", false, "enable pprof debugging server"),
flagSet.CallbackVarP(printTemplateVersion, "templates-version", "tv", "shows the version of the installed nuclei-templates"),
flagSet.BoolVarP(&options.HealthCheck, "health-check", "hc", false, "run diagnostic check up"),
@@ -412,9 +480,12 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.CreateGroup("cloud", "Cloud",
flagSet.DynamicVar(&pdcpauth, "auth", "true", "configure projectdiscovery cloud (pdcp) api key"),
- flagSet.BoolVarP(&options.EnableCloudUpload, "cloud-upload", "cup", false, "upload scan results to pdcp dashboard"),
+ flagSet.StringVarP(&options.TeamID, "team-id", "tid", _pdcp.TeamIDEnv, "upload scan results to given team id (optional)"),
+ flagSet.BoolVarP(&options.EnableCloudUpload, "cloud-upload", "cup", false, "upload scan results to pdcp dashboard [DEPRECATED use -dashboard]"),
flagSet.StringVarP(&options.ScanID, "scan-id", "sid", "", "upload scan results to existing scan id (optional)"),
flagSet.StringVarP(&options.ScanName, "scan-name", "sname", "", "scan name to set (optional)"),
+ flagSet.BoolVarP(&options.EnableCloudUpload, "dashboard", "pd", false, "upload / view nuclei results in projectdiscovery cloud (pdcp) UI dashboard"),
+ flagSet.StringVarP(&options.ScanUploadFile, "dashboard-upload", "pdu", "", "upload / view nuclei results file (jsonl) in projectdiscovery cloud (pdcp) UI dashboard"),
)
flagSet.CreateGroup("Authentication", "Authentication",
@@ -454,6 +525,11 @@ Additional documentation is available at: https://docs.nuclei.sh/getting-started
options.DAST = true
}
+ // All cloud-based templates depend on both code and self-contained templates.
+ if options.EnableCodeTemplates {
+ options.EnableSelfContainedTemplates = true
+ }
+
// api key hierarchy: cli flag > env var > .pdcp/credential file
if pdcpauth == "true" {
runner.AuthWithPDCP()
diff --git a/cmd/tmc/main.go b/cmd/tmc/main.go
index af3efd7a04..e5da46d555 100644
--- a/cmd/tmc/main.go
+++ b/cmd/tmc/main.go
@@ -69,6 +69,7 @@ func init() {
// need to set headless to true for headless templates
defaultOpts.Headless = true
defaultOpts.EnableCodeTemplates = true
+ defaultOpts.EnableSelfContainedTemplates = true
if err := protocolstate.Init(defaultOpts); err != nil {
gologger.Fatal().Msgf("Could not initialize protocol state: %s\n", err)
}
@@ -162,7 +163,7 @@ func process(opts options) error {
var updated bool // if max-requests is updated
dataString, updated, err = parseAndAddMaxRequests(templateCatalog, path, dataString)
if err != nil {
- gologger.Info().Label("max-request").Msgf(logErrMsg(path, err, opts.debug, errFile))
+ gologger.Info().Label("max-request").Msg(logErrMsg(path, err, opts.debug, errFile))
} else {
if updated {
gologger.Info().Label("max-request").Msgf("✅ updated template: %s\n", path)
@@ -255,7 +256,7 @@ func enhanceTemplate(data string) (string, bool, error) {
return data, false, errorutil.New("validation failed").WithTag("validate")
}
if templateResp.Error.Name != "" {
- return data, false, errorutil.New(templateResp.Error.Name)
+ return data, false, errorutil.New("%s", templateResp.Error.Name)
}
if strings.TrimSpace(templateResp.Enhanced) == "" && !templateResp.Lint {
if templateResp.LintError.Reason != "" {
@@ -289,7 +290,7 @@ func formatTemplate(data string) (string, bool, error) {
return data, false, errorutil.New("validation failed").WithTag("validate")
}
if templateResp.Error.Name != "" {
- return data, false, errorutil.New(templateResp.Error.Name)
+ return data, false, errorutil.New("%s", templateResp.Error.Name)
}
if strings.TrimSpace(templateResp.Updated) == "" && !templateResp.Lint {
if templateResp.LintError.Reason != "" {
@@ -300,7 +301,7 @@ func formatTemplate(data string) (string, bool, error) {
return data, false, errorutil.New("template format failed")
}
-// lintTemplateData lints template data using templateman lint api
+// lintTemplate lints template data using templateman lint api
func lintTemplate(data string) (bool, error) {
resp, err := retryablehttp.DefaultClient().Post(fmt.Sprintf("%s/lint", tmBaseUrl), "application/x-yaml", strings.NewReader(data))
if err != nil {
@@ -345,7 +346,7 @@ func validateTemplate(data string) (bool, error) {
return false, errorutil.New("validation failed").WithTag("validate")
}
if validateResp.Error.Name != "" {
- return false, errorutil.New(validateResp.Error.Name)
+ return false, errorutil.New("%s", validateResp.Error.Name)
}
return false, errorutil.New("template validation failed")
}
diff --git a/examples/advanced/advanced.go b/examples/advanced/advanced.go
index 61f1d78dd1..47e6a180d4 100644
--- a/examples/advanced/advanced.go
+++ b/examples/advanced/advanced.go
@@ -1,13 +1,28 @@
package main
import (
+<<<<<<< HEAD
nuclei "github.com/Explorer1092/nuclei/v3/lib"
+=======
+ "context"
+
+ nuclei "github.com/projectdiscovery/nuclei/v3/lib"
+ "github.com/projectdiscovery/nuclei/v3/pkg/installer"
+>>>>>>> projectdiscovery-main
syncutil "github.com/projectdiscovery/utils/sync"
)
func main() {
+ ctx := context.Background()
+ // when running nuclei in parallel for first time it is a good practice to make sure
+ // templates exists first
+ tm := installer.TemplateManager{}
+ if err := tm.FreshInstallIfNotExists(); err != nil {
+ panic(err)
+ }
+
// create nuclei engine with options
- ne, err := nuclei.NewThreadSafeNucleiEngine()
+ ne, err := nuclei.NewThreadSafeNucleiEngineCtx(ctx)
if err != nil {
panic(err)
}
diff --git a/examples/simple/simple.go b/examples/simple/simple.go
index b277f07a63..6a3a361c73 100644
--- a/examples/simple/simple.go
+++ b/examples/simple/simple.go
@@ -1,9 +1,17 @@
package main
+<<<<<<< HEAD
import nuclei "github.com/Explorer1092/nuclei/v3/lib"
+=======
+import (
+ "context"
+
+ nuclei "github.com/projectdiscovery/nuclei/v3/lib"
+)
+>>>>>>> projectdiscovery-main
func main() {
- ne, err := nuclei.NewNucleiEngine(
+ ne, err := nuclei.NewNucleiEngineCtx(context.Background(),
nuclei.WithTemplateFilters(nuclei.TemplateFilters{Tags: []string{"oast"}}),
nuclei.EnableStatsWithOpts(nuclei.StatsOptions{MetricServerPort: 6064}), // optionally enable metrics server for better observability
)
diff --git a/gh_retry.sh b/gh_retry.sh
index bf212e1290..43542276b3 100755
--- a/gh_retry.sh
+++ b/gh_retry.sh
@@ -21,9 +21,9 @@ WORKFLOW="Build Test"
GREP_ERROR_PATTERN='Test "http/interactsh.yaml" failed'
#Set fonts for Help.
-NORM=`tput sgr0`
-BOLD=`tput bold`
-REV=`tput smso`
+NORM=$(tput sgr0)
+BOLD=$(tput bold)
+REV=$(tput smso)
HELP()
{
@@ -73,7 +73,7 @@ function print_bold() {
function retry_failed_jobs() {
print_bold "Checking failed workflows for branch $BRANCH before $BEFORE"
- date=`date +%Y-%m-%d'T'%H:%M'Z' -d "$BEFORE"`
+ date=$(date +%Y-%m-%d'T'%H:%M'Z' -d "$BEFORE")
workflowIds=$(gh run list --limit "$LIMIT" --json headBranch,status,name,conclusion,databaseId,updatedAt | jq -c '.[] |
select ( .headBranch==$branch ) |
diff --git a/go.mod b/go.mod
index 9ca1040ccf..ba72a1d62e 100644
--- a/go.mod
+++ b/go.mod
@@ -1,6 +1,6 @@
module github.com/Explorer1092/nuclei/v3
-go 1.21
+go 1.22
require (
github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible
@@ -16,16 +16,16 @@ require (
github.com/json-iterator/go v1.1.12
github.com/julienschmidt/httprouter v1.3.0
github.com/logrusorgru/aurora v2.0.3+incompatible
- github.com/miekg/dns v1.1.57
+ github.com/miekg/dns v1.1.59
github.com/olekukonko/tablewriter v0.0.5
github.com/pkg/errors v0.9.1
- github.com/projectdiscovery/clistats v0.0.20
- github.com/projectdiscovery/fastdialer v0.0.69
- github.com/projectdiscovery/hmap v0.0.41
- github.com/projectdiscovery/interactsh v1.1.9
- github.com/projectdiscovery/rawhttp v0.1.47
- github.com/projectdiscovery/retryabledns v1.0.58
- github.com/projectdiscovery/retryablehttp-go v1.0.58
+ github.com/projectdiscovery/clistats v0.1.1
+ github.com/projectdiscovery/fastdialer v0.2.9
+ github.com/projectdiscovery/hmap v0.0.67
+ github.com/projectdiscovery/interactsh v1.2.0
+ github.com/projectdiscovery/rawhttp v0.1.74
+ github.com/projectdiscovery/retryabledns v1.0.85
+ github.com/projectdiscovery/retryablehttp-go v1.0.86
github.com/projectdiscovery/yamldoc-go v1.0.4
github.com/remeh/sizedwaitgroup v1.0.0
github.com/rs/xid v1.5.0
@@ -35,18 +35,18 @@ require (
github.com/spf13/cast v1.5.1
github.com/syndtr/goleveldb v1.0.0
github.com/valyala/fasttemplate v1.2.2
- github.com/weppos/publicsuffix-go v0.30.2-0.20230730094716-a20f9abcc222
- github.com/xanzy/go-gitlab v0.84.0
+ github.com/weppos/publicsuffix-go v0.30.2
+ github.com/xanzy/go-gitlab v0.107.0
go.uber.org/multierr v1.11.0
- golang.org/x/net v0.24.0
- golang.org/x/oauth2 v0.11.0
- golang.org/x/text v0.14.0
+ golang.org/x/net v0.30.0
+ golang.org/x/oauth2 v0.22.0
+ golang.org/x/text v0.19.0
gopkg.in/yaml.v2 v2.4.0
)
require (
code.gitea.io/sdk/gitea v0.17.0
- github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.3.0
+ github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.6.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.1.0
github.com/DataDog/gostackparse v0.6.0
github.com/Explorer1092/n3iwf v0.0.0-20240520140556-e3218a4830c6
@@ -59,49 +59,60 @@ require (
github.com/aws/aws-sdk-go-v2/credentials v1.13.27
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.72
github.com/aws/aws-sdk-go-v2/service/s3 v1.37.0
- github.com/charmbracelet/glamour v0.6.0
+ github.com/cespare/xxhash v1.1.0
+ github.com/charmbracelet/glamour v0.8.0
github.com/clbanning/mxj/v2 v2.7.0
- github.com/denisenkom/go-mssqldb v0.12.3
github.com/ditashi/jsbeautifier-go v0.0.0-20141206144643-2520a8026a9c
github.com/docker/go-units v0.5.0
github.com/dop251/goja v0.0.0-20240220182346-e401ed450204
github.com/fatih/structs v1.1.0
- github.com/getkin/kin-openapi v0.123.0
+ github.com/getkin/kin-openapi v0.126.0
github.com/go-git/go-git/v5 v5.11.0
github.com/go-ldap/ldap/v3 v3.4.5
github.com/go-pg/pg v8.0.7+incompatible
github.com/go-sql-driver/mysql v1.7.1
github.com/h2non/filetype v1.1.3
+ github.com/invopop/yaml v0.3.1
+ github.com/kitabisa/go-ci v1.0.3
github.com/labstack/echo/v4 v4.10.2
github.com/leslie-qiwa/flat v0.0.0-20230424180412-f9d1cf014baa
- github.com/lib/pq v1.10.1
+ github.com/lib/pq v1.10.9
github.com/mattn/go-sqlite3 v1.14.22
github.com/mholt/archiver v3.1.1+incompatible
+ github.com/microsoft/go-mssqldb v1.6.0
github.com/ory/dockertest/v3 v3.10.0
github.com/praetorian-inc/fingerprintx v1.1.9
- github.com/projectdiscovery/dsl v0.0.52
+ github.com/projectdiscovery/dsl v0.3.3
github.com/projectdiscovery/fasttemplate v0.0.2
github.com/projectdiscovery/go-smb2 v0.0.0-20240129202741-052cc450c6cb
- github.com/projectdiscovery/goflags v0.1.50
- github.com/projectdiscovery/gologger v1.1.12
+ github.com/projectdiscovery/goflags v0.1.65
+ github.com/projectdiscovery/gologger v1.1.31
github.com/projectdiscovery/gostruct v0.0.2
- github.com/projectdiscovery/gozero v0.0.2
- github.com/projectdiscovery/httpx v1.6.0
+ github.com/projectdiscovery/gozero v0.0.3
+ github.com/projectdiscovery/httpx v1.6.9
github.com/projectdiscovery/mapcidr v1.1.34
+<<<<<<< HEAD
github.com/projectdiscovery/ratelimit v0.0.39
+=======
+ github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5
+ github.com/projectdiscovery/ratelimit v0.0.61
+>>>>>>> projectdiscovery-main
github.com/projectdiscovery/rdap v0.9.1-0.20221108103045-9865884d1917
github.com/projectdiscovery/sarif v0.0.1
- github.com/projectdiscovery/tlsx v1.1.6
- github.com/projectdiscovery/uncover v1.0.7
- github.com/projectdiscovery/useragent v0.0.49
- github.com/projectdiscovery/utils v0.0.92
- github.com/projectdiscovery/wappalyzergo v0.0.120
+ github.com/projectdiscovery/tlsx v1.1.8
+ github.com/projectdiscovery/uncover v1.0.9
+ github.com/projectdiscovery/useragent v0.0.78
+ github.com/projectdiscovery/utils v0.2.18
+ github.com/projectdiscovery/wappalyzergo v0.2.2
github.com/redis/go-redis/v9 v9.1.0
github.com/seh-msft/burpxml v1.0.1
+ github.com/shurcooL/graphql v0.0.0-20230722043721-ed46e5a46466
github.com/stretchr/testify v1.9.0
- github.com/tarunKoyalwar/goleak v0.0.0-20240426214851-746d64600adc
+ github.com/tarunKoyalwar/goleak v0.0.0-20240429141123-0efa90dbdcf9
+ github.com/yassinebenaid/godump v0.10.0
github.com/zmap/zgrab2 v0.1.8-0.20230806160807-97ba87c0e706
- golang.org/x/term v0.19.0
+ go.mongodb.org/mongo-driver v1.17.0
+ golang.org/x/term v0.25.0
gopkg.in/yaml.v3 v3.0.1
moul.io/http2curl v1.0.0
)
@@ -109,13 +120,14 @@ require (
require (
aead.dev/minisign v0.2.0 // indirect
dario.cat/mergo v1.0.0 // indirect
- github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.0 // indirect
- github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 // indirect
+ github.com/Azure/azure-sdk-for-go/sdk/azcore v1.11.1 // indirect
+ github.com/Azure/azure-sdk-for-go/sdk/internal v1.8.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 // indirect
- github.com/AzureAD/microsoft-authentication-library-for-go v1.0.0 // indirect
+ github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 // indirect
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
github.com/VividCortex/ewma v1.2.0 // indirect
+ github.com/alecthomas/chroma/v2 v2.14.0 // indirect
github.com/andybalholm/brotli v1.1.0 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.27 // indirect
@@ -124,39 +136,42 @@ require (
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.4 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect
- github.com/bits-and-blooms/bitset v1.8.0 // indirect
+ github.com/bits-and-blooms/bitset v1.13.0 // indirect
github.com/bits-and-blooms/bloom/v3 v3.5.0 // indirect
github.com/buger/jsonparser v1.1.1 // indirect
github.com/bytedance/sonic v1.9.1 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
- github.com/cespare/xxhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
+ github.com/charmbracelet/lipgloss v0.13.0 // indirect
+ github.com/charmbracelet/x/ansi v0.3.2 // indirect
github.com/cheggaaa/pb/v3 v3.1.4 // indirect
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
github.com/cloudflare/cfssl v1.6.4 // indirect
- github.com/cloudflare/circl v1.3.7 // indirect
+ github.com/cloudflare/circl v1.3.8 // indirect
github.com/containerd/continuity v0.4.2 // indirect
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
github.com/davidmz/go-pageant v1.0.2 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
- github.com/dlclark/regexp2 v1.11.0 // indirect
+ github.com/dlclark/regexp2 v1.11.4 // indirect
github.com/docker/cli v24.0.5+incompatible // indirect
github.com/docker/docker v24.0.9+incompatible // indirect
github.com/docker/go-connections v0.4.0 // indirect
- github.com/fatih/color v1.15.0 // indirect
+ github.com/fatih/color v1.16.0 // indirect
github.com/free5gc/util v1.0.5-0.20230511064842-2e120956883b // indirect
github.com/gabriel-vasile/mimetype v1.4.2 // indirect
+ github.com/gaissmai/bart v0.9.5 // indirect
github.com/geoffgarside/ber v1.1.0 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/gin-gonic/gin v1.9.1 // indirect
github.com/go-asn1-ber/asn1-ber v1.5.4 // indirect
github.com/go-fed/httpsig v1.1.0 // indirect
- github.com/go-openapi/jsonpointer v0.20.2 // indirect
- github.com/go-openapi/swag v0.22.9 // indirect
+ github.com/go-openapi/jsonpointer v0.21.0 // indirect
+ github.com/go-openapi/swag v0.23.0 // indirect
github.com/go-sourcemap/sourcemap v2.1.4+incompatible // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
- github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe // indirect
+ github.com/golang-jwt/jwt/v5 v5.2.1 // indirect
+ github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
github.com/golang-sql/sqlexp v0.1.0 // indirect
github.com/google/certificate-transparency-go v1.1.4 // indirect
github.com/google/go-github/v30 v30.1.0 // indirect
@@ -166,7 +181,6 @@ require (
github.com/hashicorp/go-version v1.6.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.6 // indirect
github.com/hbakhtiyor/strsim v0.0.0-20190107154042-4d2bbb273edf // indirect
- github.com/invopop/yaml v0.2.0 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.7.6 // indirect
@@ -175,7 +189,7 @@ require (
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/kataras/jwt v0.1.10 // indirect
- github.com/klauspost/compress v1.17.6 // indirect
+ github.com/klauspost/compress v1.17.8 // indirect
github.com/klauspost/pgzip v1.2.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/logrusorgru/aurora/v4 v4.0.0 // indirect
@@ -188,32 +202,31 @@ require (
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
+ github.com/montanaflynn/stats v0.7.1 // indirect
github.com/muesli/reflow v0.3.0 // indirect
- github.com/muesli/termenv v0.15.1 // indirect
+ github.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
- github.com/opencontainers/runc v1.1.12 // indirect
+ github.com/opencontainers/runc v1.1.14 // indirect
github.com/pelletier/go-toml/v2 v2.0.8 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pjbgf/sha1cd v0.3.0 // indirect
- github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
- github.com/projectdiscovery/asnmap v1.1.0 // indirect
- github.com/projectdiscovery/cdncheck v1.0.9 // indirect
- github.com/projectdiscovery/freeport v0.0.5 // indirect
+ github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
+ github.com/projectdiscovery/asnmap v1.1.1 // indirect
+ github.com/projectdiscovery/cdncheck v1.1.0 // indirect
+ github.com/projectdiscovery/freeport v0.0.7 // indirect
github.com/projectdiscovery/ldapserver v1.0.2-0.20240219154113-dcc758ebc0cb // indirect
github.com/projectdiscovery/machineid v0.0.0-20240226150047-2e2c51e35983 // indirect
- github.com/projectdiscovery/stringsutil v0.0.2 // indirect
- github.com/quic-go/quic-go v0.42.0 // indirect
- github.com/refraction-networking/utls v1.6.1 // indirect
+ github.com/refraction-networking/utls v1.6.7 // indirect
github.com/sashabaranov/go-openai v1.15.3 // indirect
github.com/shirou/gopsutil v3.21.11+incompatible // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/skeema/knownhosts v1.2.1 // indirect
github.com/tidwall/btree v1.7.0 // indirect
- github.com/tidwall/buntdb v1.3.0 // indirect
- github.com/tidwall/gjson v1.17.0 // indirect
+ github.com/tidwall/buntdb v1.3.1 // indirect
+ github.com/tidwall/gjson v1.17.1 // indirect
github.com/tidwall/grect v0.1.4 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
@@ -223,18 +236,22 @@ require (
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.11 // indirect
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
+ github.com/xdg-go/pbkdf2 v1.0.0 // indirect
+ github.com/xdg-go/scram v1.1.2 // indirect
+ github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
+ github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
github.com/ysmood/fetchup v0.2.3 // indirect
github.com/ysmood/got v0.34.1 // indirect
- github.com/yuin/goldmark v1.5.4 // indirect
- github.com/yuin/goldmark-emoji v1.0.1 // indirect
+ github.com/yuin/goldmark v1.7.4 // indirect
+ github.com/yuin/goldmark-emoji v1.0.3 // indirect
github.com/zcalusic/sysinfo v1.0.2 // indirect
github.com/zeebo/blake3 v0.2.3 // indirect
go.uber.org/goleak v1.3.0 // indirect
golang.org/x/arch v0.3.0 // indirect
- golang.org/x/sync v0.6.0 // indirect
+ golang.org/x/sync v0.8.0 // indirect
gopkg.in/djherbis/times.v1 v1.3.0 // indirect
mellium.im/sasl v0.3.1 // indirect
)
@@ -260,15 +277,14 @@ require (
github.com/goburrow/cache v0.1.4 // indirect
github.com/gobwas/httphead v0.1.0 // indirect
github.com/gobwas/pool v0.2.1 // indirect
- github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
+ github.com/golang-jwt/jwt/v4 v4.5.1 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
- github.com/golang/protobuf v1.5.3 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/go-querystring v1.1.0 // indirect
- github.com/google/uuid v1.3.1 // indirect
+ github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/css v1.0.1 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
- github.com/hashicorp/go-retryablehttp v0.7.2 // indirect
+ github.com/hashicorp/go-retryablehttp v0.7.7 // indirect
github.com/hdm/jarm-go v0.0.7 // indirect
github.com/itchyny/timefmt-go v0.1.5 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
@@ -277,40 +293,38 @@ require (
github.com/lor00x/goldap v0.0.0-20180618054307-a546dffdd1a3 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
- github.com/mattn/go-runewidth v0.0.15 // indirect
+ github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/mholt/acmez v1.2.0 // indirect
- github.com/microcosm-cc/bluemonday v1.0.26 // indirect
+ github.com/microcosm-cc/bluemonday v1.0.27 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/projectdiscovery/blackrock v0.0.1 // indirect
- github.com/projectdiscovery/networkpolicy v0.0.8
- github.com/rivo/uniseg v0.4.6 // indirect
+ github.com/projectdiscovery/networkpolicy v0.0.9
+ github.com/rivo/uniseg v0.4.7 // indirect
github.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/trivago/tgo v1.0.7
- github.com/ulikunitz/xz v0.5.11 // indirect
+ github.com/ulikunitz/xz v0.5.12 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
- github.com/yl2chen/cidranger v1.0.2 // indirect
github.com/ysmood/goob v0.4.0 // indirect
github.com/ysmood/gson v0.7.3 // indirect
github.com/ysmood/leakless v0.8.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zmap/rc2 v0.0.0-20190804163417-abaa70531248 // indirect
- github.com/zmap/zcrypto v0.0.0-20231219022726-a1f61fb1661c // indirect
- go.etcd.io/bbolt v1.3.8 // indirect
+ github.com/zmap/zcrypto v0.0.0-20240512203510-0fef58d9a9db // indirect
+ go.etcd.io/bbolt v1.3.10 // indirect
go.uber.org/zap v1.25.0 // indirect
goftp.io/server/v2 v2.0.1 // indirect
- golang.org/x/crypto v0.22.0 // indirect
- golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
- golang.org/x/mod v0.14.0 // indirect
- golang.org/x/sys v0.19.0 // indirect
- golang.org/x/time v0.5.0 // indirect
- golang.org/x/tools v0.17.0
- google.golang.org/appengine v1.6.7 // indirect
- google.golang.org/protobuf v1.33.0 // indirect
+ golang.org/x/crypto v0.28.0 // indirect
+ golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842
+ golang.org/x/mod v0.17.0 // indirect
+ golang.org/x/sys v0.26.0 // indirect
+ golang.org/x/time v0.6.0 // indirect
+ golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d
+ google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
gopkg.in/corvus-ch/zbase32.v1 v1.0.0 // indirect
)
@@ -349,4 +363,4 @@ require (
)
// https://go.dev/ref/mod#go-mod-file-retract
-retract v3.2.0 // retract due to broken js protocol issue
+retract v3.2.0 // retract due to broken js protocol issue
\ No newline at end of file
diff --git a/go.sum b/go.sum
index 1f88017dde..db7323905e 100644
--- a/go.sum
+++ b/go.sum
@@ -40,25 +40,26 @@ dario.cat/mergo v1.0.0/go.mod h1:uNxQE+84aUszobStD9th8a29P2fMDhsBdgRYvZOxGmk=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
git.mills.io/prologic/smtpd v0.0.0-20210710122116-a525b76c287a h1:3i+FJ7IpSZHL+VAjtpQeZCRhrpP0odl5XfoLBY4fxJ8=
git.mills.io/prologic/smtpd v0.0.0-20210710122116-a525b76c287a/go.mod h1:C7hXLmFmPYPjIDGfQl1clsmQ5TMEQfmzWTrJk475bUs=
-github.com/Azure/azure-sdk-for-go/sdk/azcore v0.19.0/go.mod h1:h6H6c8enJmmocHUbLiiGY6sx7f9i+X3m1CHdd5c6Rdw=
-github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.0 h1:8q4SaHjFsClSvuVne0ID/5Ka8u3fcIHyqkLjcFpNRHQ=
-github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.0/go.mod h1:bjGvMhVMb+EEm3VRNQawDMUyMMjo+S5ewNjflkep/0Q=
-github.com/Azure/azure-sdk-for-go/sdk/azidentity v0.11.0/go.mod h1:HcM1YX14R7CJcghJGOYCgdezslRSVzqwLf/q+4Y2r/0=
-github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.3.0 h1:vcYCAze6p19qBW7MhZybIsqD8sMV8js0NyQM8JDnVtg=
-github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.3.0/go.mod h1:OQeznEEkTZ9OrhHJoDD8ZDq51FHgXjqtP9z6bEwBq9U=
-github.com/Azure/azure-sdk-for-go/sdk/internal v0.7.0/go.mod h1:yqy467j36fJxcRV2TzfVZ1pCb5vxm4BtZPUdYWe/Xo8=
-github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 h1:sXr+ck84g/ZlZUOZiNELInmMgOsuGwdjjVkEIde0OtY=
-github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0/go.mod h1:okt5dMMTOFjX/aovMlrjvvXoPMBVSPzk9185BT0+eZM=
+github.com/Azure/azure-sdk-for-go/sdk/azcore v1.11.1 h1:E+OJmp2tPvt1W+amx48v1eqbjDYsgN+RzP4q16yV5eM=
+github.com/Azure/azure-sdk-for-go/sdk/azcore v1.11.1/go.mod h1:a6xsAQUZg+VsS3TJ05SRp524Hs4pZ/AeFSr5ENf0Yjo=
+github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.6.0 h1:U2rTu3Ef+7w9FHKIAXM6ZyqF3UOWJZ12zIm8zECAFfg=
+github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.6.0/go.mod h1:9kIvujWAA58nmPmWB1m23fyWic1kYZMxD9CxaWn4Qpg=
+github.com/Azure/azure-sdk-for-go/sdk/internal v1.8.0 h1:jBQA3cKT4L2rWMpgE7Yt3Hwh2aUj8KXjIGLxjHeYNNo=
+github.com/Azure/azure-sdk-for-go/sdk/internal v1.8.0/go.mod h1:4OG6tQ9EOP/MT0NMjDlRzWoVFxfu9rN9B2X+tlSVktg=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.2.0 h1:Ma67P/GGprNwsslzEH6+Kb8nybI8jpDTm4Wmzu2ReK8=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.2.0/go.mod h1:c+Lifp3EDEamAkPVzMooRNOK6CZjNSdEnf1A7jsI9u4=
+github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.0.0 h1:yfJe15aSwEQ6Oo6J+gdfdulPNoZ3TEhmbhLIoxZcA+U=
+github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.0.0/go.mod h1:Q28U+75mpCaSCDowNEmhIo/rmgdkqmkmzI7N6TGR4UY=
+github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v0.8.0 h1:T028gtTPiYt/RMUfs8nVsAL7FDQrfLlrm/NnRG/zcC4=
+github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v0.8.0/go.mod h1:cw4zVQgBby0Z5f2v0itn6se2dDP17nTjbZFXW5uPyHA=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.1.0 h1:nVocQV40OQne5613EeLayJiRAJuKlBGy+m22qWG+WRg=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.1.0/go.mod h1:7QJP7dr2wznCMeqIrhMgWGf7XpAQnVrJqDm9nvV3Cu4=
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0=
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 h1:mFRzDkZVAjdal+s7s0MwaRv9igoPqLRdzOLzw/8Xvq8=
github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358/go.mod h1:chxPXzSsl7ZWRAuOIE23GDNzjWuZquvFlgA8xmpunjU=
-github.com/AzureAD/microsoft-authentication-library-for-go v1.0.0 h1:OBhqkivkhkMqLPymWEppkm7vgPQY2XsHoEkaMQ0AdZY=
-github.com/AzureAD/microsoft-authentication-library-for-go v1.0.0/go.mod h1:kgDmCTgBzIEPFElEF+FK0SdjAor06dRq2Go927dnQ6o=
+github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 h1:XHOnouVk1mxXfQidrMEnLlPk9UMeRtyBTnEFtxkV0kU=
+github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2/go.mod h1:wP83P5OoQ5p6ip3ScPr0BAq0BvuPAvacpEuSzyouqAI=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/DataDog/gostackparse v0.6.0 h1:egCGQviIabPwsyoWpGvIBGrEnNWez35aEO7OJ1vBI4o=
@@ -94,9 +95,15 @@ github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/
github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5/go.mod h1:SkGFH1ia65gfNATL8TAiHDNxPzPdmEL5uirI2Uyuz6c=
github.com/akrylysov/pogreb v0.10.2 h1:e6PxmeyEhWyi2AKOBIJzAEi4HkiC+lKyCocRGlnDi78=
github.com/akrylysov/pogreb v0.10.2/go.mod h1:pNs6QmpQ1UlTJKDezuRWmaqkgUE2TuU0YTWyqJZ7+lI=
+github.com/alecthomas/assert/v2 v2.7.0 h1:QtqSACNS3tF7oasA8CU6A6sXZSBDqnm7RfpLl9bZqbE=
+github.com/alecthomas/assert/v2 v2.7.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
github.com/alecthomas/chroma v0.10.0 h1:7XDcGkCQopCNKjZHfYrNLraA+M7e0fMiJ/Mfikbfjek=
github.com/alecthomas/chroma v0.10.0/go.mod h1:jtJATyUxlIORhUOFNA9NZDWGAQ8wpxQQqNSB4rjA/1s=
+github.com/alecthomas/chroma/v2 v2.14.0 h1:R3+wzpnUArGcQz7fCETQBzO5n9IMNi13iIs46aU4V9E=
+github.com/alecthomas/chroma/v2 v2.14.0/go.mod h1:QolEbTfmUHIMVpBqxeDnNBj2uoeI4EbYP4i6n68SG4I=
github.com/alecthomas/kingpin/v2 v2.3.1/go.mod h1:oYL5vtsvEHZGHxU7DMp32Dvx+qL+ptGn6lWaot2vCNE=
+github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
+github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 h1:JYp7IbQjafoB+tBA3gMyHYHrpOtNuDiK/uB5uXxq5wM=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
@@ -175,9 +182,10 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.19.3 h1:e5mnydVdCVWxP+5rPAGi2PYxC7u2
github.com/aws/aws-sdk-go-v2/service/sts v1.19.3/go.mod h1:yVGZA1CPkmUhBdA039jXNJJG7/6t+G+EBWmFq23xqnY=
github.com/aws/smithy-go v1.13.5 h1:hgz0X/DX0dGqTYpGALqXJoRKRj5oQ7150i5FdTePzO8=
github.com/aws/smithy-go v1.13.5/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
-github.com/aymanbagabas/go-osc52 v1.0.3/go.mod h1:zT8H+Rk4VSabYN90pWyugflM3ZhpTZNC7cASDfUCdT4=
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
+github.com/aymanbagabas/go-udiff v0.2.0 h1:TK0fH4MteXUDspT88n8CKzvK0X9O2xu9yQjWpi6yML8=
+github.com/aymanbagabas/go-udiff v0.2.0/go.mod h1:RE4Ex0qsGkTAJoQdQQCA0uG+nAzJO/pI/QwceO5fgrA=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
@@ -188,8 +196,9 @@ github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
-github.com/bits-and-blooms/bitset v1.8.0 h1:FD+XqgOZDUxxZ8hzoBFuV9+cGWY9CslN6d5MS5JVb4c=
github.com/bits-and-blooms/bitset v1.8.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
+github.com/bits-and-blooms/bitset v1.13.0 h1:bAQ9OPNFYbGHV6Nez0tmNI0RiEu7/hxlYJRUA0wFAVE=
+github.com/bits-and-blooms/bitset v1.13.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
github.com/bits-and-blooms/bloom/v3 v3.5.0 h1:AKDvi1V3xJCmSR6QhcBfHbCN4Vf8FfxeWkMNQfmAGhY=
github.com/bits-and-blooms/bloom/v3 v3.5.0/go.mod h1:Y8vrn7nk1tPIlmLtW2ZPV+W7StdVMor6bC1xgpjMZFs=
github.com/bluele/gcache v0.0.2 h1:WcbfdXICg7G/DGBh1PFfcirkWOQV+v077yF1pSy3DGw=
@@ -217,8 +226,14 @@ github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XL
github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/charmbracelet/glamour v0.6.0 h1:wi8fse3Y7nfcabbbDuwolqTqMQPMnVPeZhDM273bISc=
-github.com/charmbracelet/glamour v0.6.0/go.mod h1:taqWV4swIMMbWALc0m7AfE9JkPSU8om2538k9ITBxOc=
+github.com/charmbracelet/glamour v0.8.0 h1:tPrjL3aRcQbn++7t18wOpgLyl8wrOHUEDS7IZ68QtZs=
+github.com/charmbracelet/glamour v0.8.0/go.mod h1:ViRgmKkf3u5S7uakt2czJ272WSg2ZenlYEZXT2x7Bjw=
+github.com/charmbracelet/lipgloss v0.13.0 h1:4X3PPeoWEDCMvzDvGmTajSyYPcZM4+y8sCA/SsA3cjw=
+github.com/charmbracelet/lipgloss v0.13.0/go.mod h1:nw4zy0SBX/F/eAO1cWdcvy6qnkDUxr8Lw7dvFrAIbbY=
+github.com/charmbracelet/x/ansi v0.3.2 h1:wsEwgAN+C9U06l9dCVMX0/L3x7ptvY1qmjMwyfE6USY=
+github.com/charmbracelet/x/ansi v0.3.2/go.mod h1:dk73KoMTT5AX5BsX0KrqhsTqAnhZZoCBjs7dGWp4Ktw=
+github.com/charmbracelet/x/exp/golden v0.0.0-20240806155701-69247e0abc2a h1:G99klV19u0QnhiizODirwVksQB91TJKV/UaTnACcG30=
+github.com/charmbracelet/x/exp/golden v0.0.0-20240806155701-69247e0abc2a/go.mod h1:wDlXFlCrmJ8J+swcL/MnGUuYnqgQdW9rhSD61oNMb6U=
github.com/cheggaaa/pb/v3 v3.1.4 h1:DN8j4TVVdKu3WxVwcRKu0sG00IIU6FewoABZzXbRQeo=
github.com/cheggaaa/pb/v3 v3.1.4/go.mod h1:6wVjILNBaXMs8c21qRiaUM8BR82erfgau1DQ4iUXmSA=
github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06/go.mod h1:DH46F32mSOjUmXrMHnKwZdA8wcEefY7UVqBKYGjpdQY=
@@ -237,8 +252,8 @@ github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDk
github.com/cloudflare/cfssl v1.6.4 h1:NMOvfrEjFfC63K3SGXgAnFdsgkmiq4kATme5BfcqrO8=
github.com/cloudflare/cfssl v1.6.4/go.mod h1:8b3CQMxfWPAeom3zBnGJ6sd+G1NkL5TXqmDXacb+1J0=
github.com/cloudflare/circl v1.1.0/go.mod h1:prBCrKB9DV4poKZY1l9zBXg2QJY7mvgRvtMxxK7fi4I=
-github.com/cloudflare/circl v1.3.7 h1:qlCDlTPz2n9fu58M0Nh1J/JzcFpfgkFHHX3O35r5vcU=
-github.com/cloudflare/circl v1.3.7/go.mod h1:sRTcRWXGLrKw6yIGJ+l7amYJFfAXbZG0kBSc8r4zxgA=
+github.com/cloudflare/circl v1.3.8 h1:j+V8jJt09PoeMFIu2uh5JUyEaIHTXVOHslFoLNAKqwI=
+github.com/cloudflare/circl v1.3.8/go.mod h1:PDRU+oXvdD7KCtgKxW95M5Z8BpSCJXQORiZFnBQS5QU=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 h1:ox2F0PSMlrAAiAdknSRMDrAr8mfxPCfSZolH+/qQnyQ=
github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08/go.mod h1:pCxVEbcm3AMg7ejXyorUXi6HQCzOIBf7zEDVPtw0/U4=
@@ -261,8 +276,6 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davidmz/go-pageant v1.0.2 h1:bPblRCh5jGU+Uptpz6LgMZGD5hJoOt7otgT454WvHn0=
github.com/davidmz/go-pageant v1.0.2/go.mod h1:P2EDDnMqIwG5Rrp05dTRITj9z2zpGcD9efWSkTNKLIE=
-github.com/denisenkom/go-mssqldb v0.12.3 h1:pBSGx9Tq67pBOTLmxNuirNTeB8Vjmf886Kx+8Y+8shw=
-github.com/denisenkom/go-mssqldb v0.12.3/go.mod h1:k0mtMFOnU+AihqFxPMiF05rtiDrorD1Vrm1KEz5hxDo=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
@@ -273,10 +286,8 @@ github.com/ditashi/jsbeautifier-go v0.0.0-20141206144643-2520a8026a9c/go.mod h1:
github.com/dlclark/regexp2 v1.4.0/go.mod h1:2pZnwuY/m+8K6iRw6wQdMtk+rH5tNGR1i55kozfMjCc=
github.com/dlclark/regexp2 v1.4.1-0.20201116162257-a2a8dda75c91/go.mod h1:2pZnwuY/m+8K6iRw6wQdMtk+rH5tNGR1i55kozfMjCc=
github.com/dlclark/regexp2 v1.7.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
-github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
-github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
-github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
-github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
+github.com/dlclark/regexp2 v1.11.4 h1:rPYF9/LECdNymJufQKmri9gV604RvvABwgOA8un7yAo=
+github.com/dlclark/regexp2 v1.11.4/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/docker/cli v24.0.5+incompatible h1:WeBimjvS0eKdH4Ygx+ihVq1Q++xg36M/rMi4aXAvodc=
github.com/docker/cli v24.0.5+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8=
github.com/docker/docker v24.0.9+incompatible h1:HPGzNmwfLZWdxHqK9/II92pyi1EpYKsAqcl4G0Of9v0=
@@ -312,8 +323,8 @@ github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.m
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
-github.com/fatih/color v1.15.0 h1:kOqh6YHBtK8aywxGerMG2Eq3H6Qgoqeo13Bk2Mv/nBs=
-github.com/fatih/color v1.15.0/go.mod h1:0h5ZqXfHYED7Bhv2ZJamyIOUej9KtShiJESRwBDUSsw=
+github.com/fatih/color v1.16.0 h1:zmkK9Ngbjj+K0yRhTVONQh1p/HknKYSlNT+vZCzyokM=
+github.com/fatih/color v1.16.0/go.mod h1:fL2Sau1YI5c0pdGEVCbKQbLXB6edEj1ZgiY4NijnWvE=
github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
@@ -329,10 +340,12 @@ github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/gabriel-vasile/mimetype v1.4.2 h1:w5qFW6JKBz9Y393Y4q372O9A7cUSequkh1Q7OhCmWKU=
github.com/gabriel-vasile/mimetype v1.4.2/go.mod h1:zApsH/mKG4w07erKIaJPFiX0Tsq9BFQgN3qGY5GnNgA=
+github.com/gaissmai/bart v0.9.5 h1:vy+r4Px6bjZ+v2QYXAsg63vpz9IfzdW146A8Cn4GPIo=
+github.com/gaissmai/bart v0.9.5/go.mod h1:KHeYECXQiBjTzQz/om2tqn3sZF1J7hw9m6z41ftj3fg=
github.com/geoffgarside/ber v1.1.0 h1:qTmFG4jJbwiSzSXoNJeHcOprVzZ8Ulde2Rrrifu5U9w=
github.com/geoffgarside/ber v1.1.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
-github.com/getkin/kin-openapi v0.123.0 h1:zIik0mRwFNLyvtXK274Q6ut+dPh6nlxBp0x7mNrPhs8=
-github.com/getkin/kin-openapi v0.123.0/go.mod h1:wb1aSZA/iWmorQP9KTAS/phLj/t17B5jT7+fS8ed9NM=
+github.com/getkin/kin-openapi v0.126.0 h1:c2cSgLnAsS0xYfKsgt5oBV6MYRM/giU8/RtwUY4wyfY=
+github.com/getkin/kin-openapi v0.126.0/go.mod h1:7mONz8IwmSRg6RttPu6v8U/OJ+gr+J99qSFNjPGSQqw=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
@@ -369,14 +382,12 @@ github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
-github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ=
-github.com/go-logr/logr v1.2.4/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
-github.com/go-openapi/jsonpointer v0.20.2 h1:mQc3nmndL8ZBzStEo3JYF8wzmeWffDH4VbXz58sAx6Q=
-github.com/go-openapi/jsonpointer v0.20.2/go.mod h1:bHen+N0u1KEO3YlmqOjTT9Adn1RfD91Ar825/PuiRVs=
-github.com/go-openapi/swag v0.22.9 h1:XX2DssF+mQKM2DHsbgZK74y/zj4mo9I99+89xUmuZCE=
-github.com/go-openapi/swag v0.22.9/go.mod h1:3/OXnFfnMAwBD099SwYRk7GD3xOrr1iL7d/XNLXVVwE=
+github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ=
+github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
+github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+GrE=
+github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
github.com/go-pg/pg v8.0.7+incompatible h1:ty/sXL1OZLo+47KK9N8llRcmbA9tZasqbQ/OO4ld53g=
github.com/go-pg/pg v8.0.7+incompatible/go.mod h1:a2oXow+aFOrvwcKs3eIA0lNFmMilrxK2sOkB5NWe0vA=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
@@ -396,8 +407,6 @@ github.com/go-sql-driver/mysql v1.4.0/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG
github.com/go-sql-driver/mysql v1.7.1 h1:lUIinVbN1DY0xBg0eMOzmmtGoHwWBbvnWubQUrtU8EI=
github.com/go-sql-driver/mysql v1.7.1/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
-github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI=
-github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls=
github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM=
github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/goburrow/cache v0.1.4 h1:As4KzO3hgmzPlnaMniZU9+VmoNYseUhuELbxy9mRBfw=
@@ -420,10 +429,12 @@ github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69
github.com/golang-jwt/jwt v3.2.2+incompatible h1:IfV12K8xAKAnZqdXVzCZ+TOjboZ2keLg81eXfW3O+oY=
github.com/golang-jwt/jwt v3.2.2+incompatible/go.mod h1:8pz2t5EyA70fFQQSrl6XZXzqecmYZeUEB8OUGHkxJ+I=
github.com/golang-jwt/jwt/v4 v4.4.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
-github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg=
-github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
-github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe h1:lXe2qZdvpiX5WZkZR4hgp4KJVfY3nMkvmwbVkpv1rVY=
-github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
+github.com/golang-jwt/jwt/v4 v4.5.1 h1:JdqV9zKUdtaa9gdPlywC3aeoEsR681PlKC+4F5gQgeo=
+github.com/golang-jwt/jwt/v4 v4.5.1/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
+github.com/golang-jwt/jwt/v5 v5.2.1 h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
+github.com/golang-jwt/jwt/v5 v5.2.1/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
+github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 h1:au07oEsX2xN0ktxqI+Sida1w446QrXBRJ0nee3SNZlA=
+github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
github.com/golang-sql/sqlexp v0.1.0 h1:ZCD6MBpcuOVfGVqsEmY5/4FtYiKz6tSyUv9LPEDei6A=
github.com/golang-sql/sqlexp v0.1.0/go.mod h1:J4ad9Vo8ZCWQ2GMrC4UCQy1JpCbwU9m3EOqtpKwwwHI=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
@@ -456,7 +467,6 @@ github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
-github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.2/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
@@ -506,14 +516,13 @@ github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm4
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ=
github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
-github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4=
-github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gorilla/context v1.1.1/go.mod h1:kBGZzfjB9CEq2AlWe17Uuf7NDRt0dE0s8S51q0aT7Yg=
-github.com/gorilla/css v1.0.0/go.mod h1:Dn721qIggHpt4+EFCcTLTU/vk5ySda2ReITrtgBl60c=
github.com/gorilla/css v1.0.1 h1:ntNaBIghp6JmvWnxbZKANoLyuXTPZ4cAMlo6RyhlbO8=
github.com/gorilla/css v1.0.1/go.mod h1:BvnYkspnSzMmwRK+b8/xgNPLiIuNZr6vbZBTPQ2A3b0=
github.com/gorilla/mux v1.6.2/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
@@ -532,13 +541,13 @@ github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brv
github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ=
github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48=
-github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI=
-github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
+github.com/hashicorp/go-hclog v1.6.3 h1:Qr2kF+eVWjTiYmU7Y31tYlP1h0q/X3Nl3tPGdaB11/k=
+github.com/hashicorp/go-hclog v1.6.3/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M=
github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
github.com/hashicorp/go-msgpack v0.5.3/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM=
github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk=
-github.com/hashicorp/go-retryablehttp v0.7.2 h1:AcYqCvkpalPnPF2pn0KamgwamS42TqUDDYFRKq/RAd0=
-github.com/hashicorp/go-retryablehttp v0.7.2/go.mod h1:Jy/gPYAdjqffZ/yFGCFV2doI5wjtH1ewM9u8iYVjtX8=
+github.com/hashicorp/go-retryablehttp v0.7.7 h1:C8hUCYzor8PIfXHa4UrZkU4VvK8o9ISHxT2Q8+VepXU=
+github.com/hashicorp/go-retryablehttp v0.7.7/go.mod h1:pkQpWZeYWskR+D1tR2O5OcBFOxfA7DoAO6xtkuQnHTk=
github.com/hashicorp/go-rootcerts v1.0.0/go.mod h1:K6zTfqpRlCUIjkwsN4Z+hiSfzSTQa6eBIzfwKfwNnHU=
github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU=
github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdvsLplgctolz4=
@@ -564,6 +573,8 @@ github.com/hbakhtiyor/strsim v0.0.0-20190107154042-4d2bbb273edf h1:umfGUaWdFP2s6
github.com/hbakhtiyor/strsim v0.0.0-20190107154042-4d2bbb273edf/go.mod h1:V99KdStnMHZsvVOwIvhfcUzYgYkRZeQWUtumtL+SKxA=
github.com/hdm/jarm-go v0.0.7 h1:Eq0geenHrBSYuKrdVhrBdMMzOmA+CAMLzN2WrF3eL6A=
github.com/hdm/jarm-go v0.0.7/go.mod h1:kinGoS0+Sdn1Rr54OtanET5E5n7AlD6T6CrJAKDjJSQ=
+github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
+github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/hudl/fargo v1.3.0/go.mod h1:y3CKSmjA+wD2gak7sUSXTAoopbhU08POFhmITJgmKTg=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
@@ -575,8 +586,8 @@ github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANyt
github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d/go.mod h1:qj24IKcXYK6Iy9ceXlo3Tc+vtHo9lIhSX5JddghvEPo=
github.com/invopop/jsonschema v0.12.0 h1:6ovsNSuvn9wEQVOyc72aycBMVQFKz7cPdMJn10CvzRI=
github.com/invopop/jsonschema v0.12.0/go.mod h1:ffZ5Km5SWWRAIN6wbDXItl95euhFz2uON45H2qjYt+0=
-github.com/invopop/yaml v0.2.0 h1:7zky/qH+O0DwAyoobXUqvVBwgBFRxKoQ/3FjcVpjTMY=
-github.com/invopop/yaml v0.2.0/go.mod h1:2XuRLgs/ouIrW3XNzuNj7J3Nvu/Dig5MXvbCEdiBN3Q=
+github.com/invopop/yaml v0.3.1 h1:f0+ZpmhfBSS4MhG+4HYseMdJhoeeopbSKbq5Rpeelso=
+github.com/invopop/yaml v0.3.1/go.mod h1:PMOp3nn4/12yEZUFfmOuNHJsZToEEOwoWsT+D81KkeA=
github.com/itchyny/gojq v0.12.13 h1:IxyYlHYIlspQHHTE0f3cJF0NKDMfajxViuhBLnHd/QU=
github.com/itchyny/gojq v0.12.13/go.mod h1:JzwzAqenfhrPUuwbmEz3nu3JQmFLlQTQMUcOdnu/Sf4=
github.com/itchyny/timefmt-go v0.1.5 h1:G0INE2la8S6ru/ZI5JecgyzbbJNs5lG1RcBqa7Jm6GE=
@@ -630,10 +641,12 @@ github.com/kevinburke/ssh_config v1.2.0/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/kitabisa/go-ci v1.0.3 h1:JmIUIvcercRQc/9x/v02ydCCqU4MadSHaNaOF8T2pGA=
+github.com/kitabisa/go-ci v1.0.3/go.mod h1:e3wBSzaJbcifXrr/Gw2ZBLn44MmeqP5WySwXyHlCK/U=
github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/compress v1.11.4/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
-github.com/klauspost/compress v1.17.6 h1:60eq2E/jlfwQXtvZEeBUYADs+BwKBWURIY+Gj2eRGjI=
-github.com/klauspost/compress v1.17.6/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
+github.com/klauspost/compress v1.17.8 h1:YcnTYrq7MikUT7k0Yb5eceMmALQPYBW/Xltxn0NAMnU=
+github.com/klauspost/compress v1.17.8/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
@@ -664,8 +677,8 @@ github.com/leodido/go-urn v1.2.4 h1:XlAE/cm/ms7TE/VMVoduSpNBoyc2dOxHs5MZSwAN63Q=
github.com/leodido/go-urn v1.2.4/go.mod h1:7ZrI8mTSeBSHl/UaRyKQW1qZeMgak41ANeCNaVckg+4=
github.com/leslie-qiwa/flat v0.0.0-20230424180412-f9d1cf014baa h1:KQKuQDgA3DZX6C396lt3WDYB9Um1gLITLbvficVbqXk=
github.com/leslie-qiwa/flat v0.0.0-20230424180412-f9d1cf014baa/go.mod h1:HbwNE4XGwjgtUELkvQaAOjWrpianHYZdQVNqSdYW3UM=
-github.com/lib/pq v1.10.1 h1:6VXZrLU0jHBYyAqrSPa+MgPfnSvTPuMgK+k0o5kVFWo=
-github.com/lib/pq v1.10.1/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
+github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
+github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/libdns/libdns v0.2.1 h1:Wu59T7wSHRgtA0cfxC+n1c/e+O3upJGWytknkmFEDis=
github.com/libdns/libdns v0.2.1/go.mod h1:yQCXzk1lEZmmCPa857bnk4TsOiqYasqpyOEeSObbb40=
github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-bc2310a04743/go.mod h1:qklhhLq1aX+mtWk9cPHPzaBjWImj5ULL6C7HFJtXQMM=
@@ -698,9 +711,8 @@ github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D
github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.12/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
-github.com/mattn/go-runewidth v0.0.14/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
-github.com/mattn/go-runewidth v0.0.15 h1:UNAjwbU9l54TA3KzvqLGxwWjHmMgBUVhBiTjelZgg3U=
-github.com/mattn/go-runewidth v0.0.15/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
+github.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc=
+github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU=
github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
@@ -711,13 +723,14 @@ github.com/mholt/archiver v3.1.1+incompatible h1:1dCVxuqs0dJseYEhi5pl7MYPH9zDa1w
github.com/mholt/archiver v3.1.1+incompatible/go.mod h1:Dh2dOXnSdiLxRiPoVfIr/fI1TwETms9B8CTWfeh7ROU=
github.com/mholt/archiver/v3 v3.5.1 h1:rDjOBX9JSF5BvoJGvjqK479aL70qh9DIpZCl+k7Clwo=
github.com/mholt/archiver/v3 v3.5.1/go.mod h1:e3dqJ7H78uzsRSEACH1joayhuSyhnonssnDhppzS1L4=
-github.com/microcosm-cc/bluemonday v1.0.21/go.mod h1:ytNkv4RrDrLJ2pqlsSI46O6IVXmZOBBD4SaJyDwwTkM=
-github.com/microcosm-cc/bluemonday v1.0.26 h1:xbqSvqzQMeEHCqMi64VAs4d8uy6Mequs3rQ0k/Khz58=
-github.com/microcosm-cc/bluemonday v1.0.26/go.mod h1:JyzOCs9gkyQyjs+6h10UEVSe02CGwkhd72Xdqh78TWs=
+github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
+github.com/microcosm-cc/bluemonday v1.0.27/go.mod h1:jFi9vgW+H7c3V0lb6nR74Ib/DIB5OBs92Dimizgw2cA=
+github.com/microsoft/go-mssqldb v1.6.0 h1:mM3gYdVwEPFrlg/Dvr2DNVEgYFG7L42l+dGc67NNNpc=
+github.com/microsoft/go-mssqldb v1.6.0/go.mod h1:00mDtPbeQCRGC1HwOOR5K/gr30P1NcEG0vx6Kbv2aJU=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
github.com/miekg/dns v1.1.35/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM=
-github.com/miekg/dns v1.1.57 h1:Jzi7ApEIzwEPLHWRcafCN9LZSBbqQpxjt/wpgvg7wcM=
-github.com/miekg/dns v1.1.57/go.mod h1:uqRjCRUuEAA6qsOiJvDd+CFo/vW+y5WR6SNmHE55hZk=
+github.com/miekg/dns v1.1.59 h1:C9EXc/UToRwKLhK5wKU/I4QVsBUc8kE6MkHBkeypWZs=
+github.com/miekg/dns v1.1.59/go.mod h1:nZpewl5p6IvctfgrckopVx2OlSEHPRO/U4SYkRklrEk=
github.com/minio/minio-go/v6 v6.0.46/go.mod h1:qD0lajrGW49lKZLtXKtCB4X/qkMf0a5tBvN2PaZg7Gg=
github.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7 h1:yRZGarbxsRytL6EGgbqK2mCY+Lk5MWKQYKJT2gEglhc=
github.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7/go.mod h1:bO02GTIPCMQFTEvE5h4DjYB58bCoZ35XLeBf0buTDdM=
@@ -742,16 +755,16 @@ github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lN
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
-github.com/modocache/gover v0.0.0-20171022184752-b58185e213c5/go.mod h1:caMODM3PzxT8aQXRPkAt8xlV/e7d7w8GM5g0fa5F0D8=
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 h1:RWengNIwukTxcDr9M+97sNutRR1RKhG96O6jWumTTnw=
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826/go.mod h1:TaXosZuwdSHYgviHp1DAtfrULt5eUgsSMsZf+YrPgl8=
+github.com/montanaflynn/stats v0.7.1 h1:etflOAAHORrCC44V+aR6Ftzort912ZU+YLiSTuV8eaE=
+github.com/montanaflynn/stats v0.7.1/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow=
github.com/mreiferson/go-httpclient v0.0.0-20160630210159-31f0106b4474/go.mod h1:OQA4XLvDbMgS8P0CevmM4m9Q3Jq4phKUzcocxuGJ5m8=
github.com/mreiferson/go-httpclient v0.0.0-20201222173833-5e475fde3a4d/go.mod h1:OQA4XLvDbMgS8P0CevmM4m9Q3Jq4phKUzcocxuGJ5m8=
github.com/muesli/reflow v0.3.0 h1:IFsN6K9NfGtjeggFP+68I4chLZV2yIKsXJFNZ+eWh6s=
github.com/muesli/reflow v0.3.0/go.mod h1:pbwTDkVPibjO2kyvBQRBxTWEEGDGq0FlB1BIKtnHY/8=
-github.com/muesli/termenv v0.13.0/go.mod h1:sP1+uffeLaEYpyOTb8pLCUctGcGLnoFjSn4YJK5e2bc=
-github.com/muesli/termenv v0.15.1 h1:UzuTb/+hhlBugQz28rpzey4ZuKcZ03MeKsoG7IJZIxs=
-github.com/muesli/termenv v0.15.1/go.mod h1:HeAQPTzpfs016yGtA4g00CsdYnVLJvxsS4ANqrZs2sQ=
+github.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a h1:2MaM6YC3mGu54x+RKAA6JiFFHlHDY1UbkxqppT7wYOg=
+github.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a/go.mod h1:hxSnBBYLK21Vtq/PHd0S2FYCxBXzBua8ov5s1RobyRQ=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/nats-io/jwt v0.3.0/go.mod h1:fRYCDE99xlTsqUzISS1Bi75UBJ6ljOJQOAAu5VglpSg=
@@ -775,8 +788,6 @@ github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.16.4 h1:29JGrr5oVBm5ulCWet69zQkzWipVXIol6ygQUe/EzNc=
github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
-github.com/onsi/ginkgo/v2 v2.9.5 h1:+6Hr4uxzP4XIUyAkg61dWBw8lb/gc4/X5luuxN/EC+Q=
-github.com/onsi/ginkgo/v2 v2.9.5/go.mod h1:tvAoo1QUJwNEU2ITftXTpR7R1RbCzoZUOs3RonqW57k=
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.27.10 h1:naR28SdDFlqrG6kScpT8VWpu1xWY5nJRCF3XaYyBjhI=
github.com/onsi/gomega v1.27.10/go.mod h1:RsS8tutOdbdgzbPtzzATp12yT7kM5I5aElG3evPbQ0M=
@@ -785,8 +796,8 @@ github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.0.2 h1:9yCKha/T5XdGtO0q9Q9a6T5NUCsTn/DrBg0D7ufOcFM=
github.com/opencontainers/image-spec v1.0.2/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
-github.com/opencontainers/runc v1.1.12 h1:BOIssBaW1La0/qbNZHXOOa71dZfZEQOzW7dqQf3phss=
-github.com/opencontainers/runc v1.1.12/go.mod h1:S+lQwSfncpBha7XTy/5lBwWgm5+y5Ma/O44Ekby9FK8=
+github.com/opencontainers/runc v1.1.14 h1:rgSuzbmgz5DUJjeSnw337TxDbRuqjs6iqQck/2weR6w=
+github.com/opencontainers/runc v1.1.14/go.mod h1:E4C2z+7BxR7GHXp0hAY53mek+x49X1LjPNeMTfRGvOA=
github.com/opentracing-contrib/go-observer v0.0.0-20170622124052-a52f23424492/go.mod h1:Ngi6UdF0k5OKD5t5wlmGhe/EDKPoUM3BXZSSfIuJbis=
github.com/opentracing/basictracer-go v1.0.0/go.mod h1:QfBfYuafItcjQuMwinw9GhYKwFXS9KnPs5lxoYwgW74=
github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
@@ -814,9 +825,8 @@ github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ
github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pjbgf/sha1cd v0.3.0 h1:4D5XXmUUBUl/xQ6IjCkEAbqXskkq/4O7LmGn0AqMDs4=
github.com/pjbgf/sha1cd v0.3.0/go.mod h1:nZ1rrWOcGJ5uZgEEVL1VUM9iRQiZvWdbZjkKyFzPPsI=
-github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4/go.mod h1:4OwLy04Bl9Ef3GJJCoec+30X3LQs/0/m4HFRt/2LUSA=
-github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 h1:KoWmjvw+nsYOo29YJK9vDA65RGE3NrOnUtO7a+RF9HU=
-github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8/go.mod h1:HKlIX3XHQyzLZPlr7++PzdhaXEj94dEiJgZDTsxEqUI=
+github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
+github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
@@ -829,70 +839,81 @@ github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c h1:ncq/mPwQF
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/praetorian-inc/fingerprintx v1.1.9 h1:zWbG/Fdan0s/dvXkeaHb/CdFTz/yEEzrAF4iCzok3r8=
github.com/praetorian-inc/fingerprintx v1.1.9/go.mod h1:k6EJIHe/Da4DH5e4JuoZHe+qSGq/KPUmXGaK+xW74OI=
-github.com/projectdiscovery/asnmap v1.1.0 h1:ynvbLB5cNpyQ2+k9IP0Rpla+0JmCJpd3mw6KLAW13m0=
-github.com/projectdiscovery/asnmap v1.1.0/go.mod h1:QNjBnGLxUBEZAgaYk/Av5cjKKWFY3i/FOfoIWCUApoY=
+github.com/projectdiscovery/asnmap v1.1.1 h1:ImJiKIaACOT7HPx4Pabb5dksolzaFYsD1kID2iwsDqI=
+github.com/projectdiscovery/asnmap v1.1.1/go.mod h1:QT7jt9nQanj+Ucjr9BqGr1Q2veCCKSAVyUzLXfEcQ60=
github.com/projectdiscovery/blackrock v0.0.1 h1:lHQqhaaEFjgf5WkuItbpeCZv2DUIE45k0VbGJyft6LQ=
github.com/projectdiscovery/blackrock v0.0.1/go.mod h1:ANUtjDfaVrqB453bzToU+YB4cUbvBRpLvEwoWIwlTss=
-github.com/projectdiscovery/cdncheck v1.0.9 h1:BS15gzj9gb5AVSKqTDzPamfSgStu7nJQOocUvrssFlg=
-github.com/projectdiscovery/cdncheck v1.0.9/go.mod h1:18SSl1w7rMj53CGeRIZTbDoa286a6xZIxGbaiEo4Fxs=
-github.com/projectdiscovery/clistats v0.0.20 h1:5jO5SLiRJ7f0nDV0ndBNmBeesbROouPooH+DGMgoWq4=
-github.com/projectdiscovery/clistats v0.0.20/go.mod h1:GJ2av0KnOvK0AISQnP8hyDclYIji1LVkx2l0pwnzAu4=
-github.com/projectdiscovery/dsl v0.0.52 h1:jvIvF+qN8+MbI1MHtWJJKfWqAZQlCExL3ob7SddQbZE=
-github.com/projectdiscovery/dsl v0.0.52/go.mod h1:xfcHwhy2HSaeGgh+1wqzOoCGm2XTdh5JzjBRBVHEMvI=
-github.com/projectdiscovery/fastdialer v0.0.69 h1:BfFQTyTB1hrw9sWCw4CjQfbmlpvnJCPZEmKtxcwJGbU=
-github.com/projectdiscovery/fastdialer v0.0.69/go.mod h1:RXrx7M2T3V3rMZ2h0X2/SsY93+RhgF/LmFa1E0MI3L8=
+github.com/projectdiscovery/cdncheck v1.1.0 h1:qDITidmJsejzpk3rMkauCh6sjI2GH9hW/snk0cQ3kXE=
+github.com/projectdiscovery/cdncheck v1.1.0/go.mod h1:sZ8U4MjHSsyaTVjBbYWHT1cwUVvUYwDX1W+WvWRicIc=
+github.com/projectdiscovery/clistats v0.1.1 h1:8mwbdbwTU4aT88TJvwIzTpiNeow3XnAB72JIg66c8wE=
+github.com/projectdiscovery/clistats v0.1.1/go.mod h1:4LtTC9Oy//RiuT1+76MfTg8Hqs7FQp1JIGBM3nHK6a0=
+github.com/projectdiscovery/dsl v0.3.3 h1:4Ij5S86cHlb6xFrS7+5zAiJPeBt5h970XBTHqeTkpyU=
+github.com/projectdiscovery/dsl v0.3.3/go.mod h1:DAjSeaogLM9f0Ves2zDc/vbJrfcv+kEmS51p0dLLaPI=
+github.com/projectdiscovery/fastdialer v0.2.9 h1:vDCqxVMCyUu3oVEizEK1K8K+CCcLkVDW3X2HfiWaVFA=
+github.com/projectdiscovery/fastdialer v0.2.9/go.mod h1:mYv5QaNBDDSHlZO9DI0niRMw+G5hUzwIhs8QixSElUI=
github.com/projectdiscovery/fasttemplate v0.0.2 h1:h2cISk5xDhlJEinlBQS6RRx0vOlOirB2y3Yu4PJzpiA=
github.com/projectdiscovery/fasttemplate v0.0.2/go.mod h1:XYWWVMxnItd+r0GbjA1GCsUopMw1/XusuQxdyAIHMCw=
-github.com/projectdiscovery/freeport v0.0.5 h1:jnd3Oqsl4S8n0KuFkE5Hm8WGDP24ITBvmyw5pFTHS8Q=
-github.com/projectdiscovery/freeport v0.0.5/go.mod h1:PY0bxSJ34HVy67LHIeF3uIutiCSDwOqKD8ruBkdiCwE=
+github.com/projectdiscovery/freeport v0.0.7 h1:Q6uXo/j8SaV/GlAHkEYQi8WQoPXyJWxyspx+aFmz9Qk=
+github.com/projectdiscovery/freeport v0.0.7/go.mod h1:cOhWKvNBe9xM6dFJ3RrrLvJ5vXx2NQ36SecuwjenV2k=
github.com/projectdiscovery/go-smb2 v0.0.0-20240129202741-052cc450c6cb h1:rutG906Drtbpz4DwU5mhGIeOhRcktDH4cGQitGUMAsg=
github.com/projectdiscovery/go-smb2 v0.0.0-20240129202741-052cc450c6cb/go.mod h1:FLjF1DmZ+POoGEiIQdWuYVwS++C/GwpX8YaCsTSm1RY=
-github.com/projectdiscovery/goflags v0.1.50 h1:vhpM3mDTnXTktypUuVKWxgRpuBFJE+wzOr56rnOl4Bc=
-github.com/projectdiscovery/goflags v0.1.50/go.mod h1:3A8u7Q7e59S7f6QlJmMvKMXD/Tp+bfWre64gUpt6a6Q=
-github.com/projectdiscovery/gologger v1.1.12 h1:uX/QkQdip4PubJjjG0+uk5DtyAi1ANPJUvpmimXqv4A=
-github.com/projectdiscovery/gologger v1.1.12/go.mod h1:DI8nywPLERS5mo8QEA9E7gd5HZ3Je14SjJBH3F5/kLw=
+github.com/projectdiscovery/goflags v0.1.65 h1:rjoj+5lP/FDzgeM0WILUTX9AOOnw0J0LXtl8P1SVeGE=
+github.com/projectdiscovery/goflags v0.1.65/go.mod h1:cg6+yrLlaekP1hnefBc/UXbH1YGWa0fuzEW9iS1aG4g=
+github.com/projectdiscovery/gologger v1.1.31 h1:FlZi1RsDoRtOkj9+a1PhcOmwD3NdRpDyjp/0/fmpQ/s=
+github.com/projectdiscovery/gologger v1.1.31/go.mod h1:zVbkxOmWuh1GEyr6dviEPNwH/GMWdnJrMUSOJbRmDqI=
github.com/projectdiscovery/gostruct v0.0.2 h1:s8gP8ApugGM4go1pA+sVlPDXaWqNP5BBDDSv7VEdG1M=
github.com/projectdiscovery/gostruct v0.0.2/go.mod h1:H86peL4HKwMXcQQtEa6lmC8FuD9XFt6gkNR0B/Mu5PE=
-github.com/projectdiscovery/gozero v0.0.2 h1:8fJeaCjxL9tpm33uG/RsCQs6HGM/NE6eA3cjkilRQ+E=
-github.com/projectdiscovery/gozero v0.0.2/go.mod h1:d8bZvDWW07LWNYWrwjZ4OO1I0cpkfqaysyDfSs9ibK8=
-github.com/projectdiscovery/hmap v0.0.41 h1:8IgTyDce3/2JzcfPVA4H+XpBRFfETULx8td3BMdSYVE=
-github.com/projectdiscovery/hmap v0.0.41/go.mod h1:bCrai6x5Eijqm2U+jtcH0wZX5ZcaZhcvzoMGTZgLAf0=
-github.com/projectdiscovery/httpx v1.6.0 h1:6g4UoSQpsOyZgaK+SMLLnZIAU0eYyTxBUwVl+jtm0JQ=
-github.com/projectdiscovery/httpx v1.6.0/go.mod h1:dzMzOWKfeofaXcXzac3O+YmuY24P0CRnviKGxvol6MM=
-github.com/projectdiscovery/interactsh v1.1.9 h1:b77SaSGrO+DtivmWwqGGY2dmNlQC3Zgmwlaj9L4Oqvc=
-github.com/projectdiscovery/interactsh v1.1.9/go.mod h1:0FRQXCildcTLq7Tsa4BVZAsFCXhpWs4C4quKWigXb5I=
+github.com/projectdiscovery/gozero v0.0.3 h1:tsYkrSvWw4WdIUJyisd4MB1vRiw1X57TuVVk3p8Z3G8=
+github.com/projectdiscovery/gozero v0.0.3/go.mod h1:MpJ37Dsh94gy2EKqaemdeh+CzduGVB2SDfhr6Upsjew=
+github.com/projectdiscovery/hmap v0.0.67 h1:PG09AyXH6mchdZCdxAS7WkZz0xxsOsIxJOmEixEmnzI=
+github.com/projectdiscovery/hmap v0.0.67/go.mod h1:WxK8i2J+wcdimIXCgpYzfj9gKxCqRqOM4KENDRzGgAA=
+github.com/projectdiscovery/httpx v1.6.9 h1:ihyFclesLjvQpiJpRIlAYeebapyIbOI/arDAvvy1ES8=
+github.com/projectdiscovery/httpx v1.6.9/go.mod h1:zQtX5CtcDYXzIRWne1ztCVtqG0sXCnx84tFwfMHoB8Q=
+github.com/projectdiscovery/interactsh v1.2.0 h1:Al6jHiR+Usl9egYJDLJaWNHOcH8Rugk8gWMasc8Cmw8=
+github.com/projectdiscovery/interactsh v1.2.0/go.mod h1:Wxt0fnzxsfrAZQQlpVrf3xMatP4OXZaZbjuDkIQKdYY=
github.com/projectdiscovery/ldapserver v1.0.2-0.20240219154113-dcc758ebc0cb h1:MGtI4oE12ruWv11ZlPXXd7hl/uAaQZrFvrIDYDeVMd8=
github.com/projectdiscovery/ldapserver v1.0.2-0.20240219154113-dcc758ebc0cb/go.mod h1:vmgC0DTFCfoCLp0RAfsfYTZZan0QMVs+cmTbH6blfjk=
github.com/projectdiscovery/machineid v0.0.0-20240226150047-2e2c51e35983 h1:ZScLodGSezQVwsQDtBSMFp72WDq0nNN+KE/5DHKY5QE=
github.com/projectdiscovery/machineid v0.0.0-20240226150047-2e2c51e35983/go.mod h1:3G3BRKui7nMuDFAZKR/M2hiOLtaOmyukT20g88qRQjI=
github.com/projectdiscovery/mapcidr v1.1.34 h1:udr83vQ7oz3kEOwlsU6NC6o08leJzSDQtls1wmXN/kM=
github.com/projectdiscovery/mapcidr v1.1.34/go.mod h1:1+1R6OkKSAKtWDXE9RvxXtXPoajXTYX0eiEdkqlhQqQ=
+<<<<<<< HEAD
github.com/projectdiscovery/networkpolicy v0.0.8 h1:XvfBaBwSDNTesSfNQP9VLk3HX9I7x7gHm028TJ5XwI8=
github.com/projectdiscovery/networkpolicy v0.0.8/go.mod h1:xnjNqhemxUPxU+UD5Jgsc3+K8IVmcqT1SJeo6UzMtkI=
github.com/projectdiscovery/ratelimit v0.0.39 h1:gMpkkuRSzsxNOxPOiD0aYTLQTOiqM/I0WZSZ7kmL+hc=
github.com/projectdiscovery/ratelimit v0.0.39/go.mod h1:vR+q959CbyXCV/pD4bl4aGP2/R91BOKv1SyiHDOqeL0=
github.com/projectdiscovery/rawhttp v0.1.47 h1:bjhZBK+7iuvcu0QTRJjdS4wP747+33Sd5gfgc4V9AWU=
github.com/projectdiscovery/rawhttp v0.1.47/go.mod h1:2XA7xODWEGoHpEB8qzgqsV8yVL3cg5G63ZaT9ALwU4g=
+=======
+github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5 h1:L/e8z8yw1pfT6bg35NiN7yd1XKtJap5Nk6lMwQ0RNi8=
+github.com/projectdiscovery/n3iwf v0.0.0-20230523120440-b8cd232ff1f5/go.mod h1:pGW2ncnTxTxHtP9wzcIJAB+3/NMp6IiuQWd2NK7K+oc=
+github.com/projectdiscovery/networkpolicy v0.0.9 h1:IrlDoYZagNNO8y+7iZeHT8k5izE+nek7TdtvEBwCxqk=
+github.com/projectdiscovery/networkpolicy v0.0.9/go.mod h1:XFJ2Lnv8BE/ziQCFjBHMsH1w6VmkPiQtk+NlBpdMU7M=
+github.com/projectdiscovery/ratelimit v0.0.61 h1:n9PD4Z4Y6cLeT2rn9IiOAA0I/kIZE/D7z7z5X/WQds8=
+github.com/projectdiscovery/ratelimit v0.0.61/go.mod h1:u7DxBBcUzFg4Cb2s5yabmtCMJs+ojulNpNrSLtftoKg=
+github.com/projectdiscovery/rawhttp v0.1.74 h1:ahE23GwPyFDBSofmo92MuW439P4x20GBYwOFqejY5G8=
+github.com/projectdiscovery/rawhttp v0.1.74/go.mod h1:xEqBY17CHgGmMfuLOWYntjFQ9crb4PG1xoNgexcAq4g=
+>>>>>>> projectdiscovery-main
github.com/projectdiscovery/rdap v0.9.1-0.20221108103045-9865884d1917 h1:m03X4gBVSorSzvmm0bFa7gDV4QNSOWPL/fgZ4kTXBxk=
github.com/projectdiscovery/rdap v0.9.1-0.20221108103045-9865884d1917/go.mod h1:JxXtZC9e195awe7EynrcnBJmFoad/BNDzW9mzFkK8Sg=
-github.com/projectdiscovery/retryabledns v1.0.58 h1:ut1FSB9+GZ6zQIlKJFLqIz2RZs81EmkbsHTuIrWfYLE=
-github.com/projectdiscovery/retryabledns v1.0.58/go.mod h1:RobmKoNBgngAVE4H9REQtaLP1pa4TCyypHy1MWHT1mY=
-github.com/projectdiscovery/retryablehttp-go v1.0.58 h1:i5BlSJGgNnoTULyqLSe3d39o/XShxK4oEvx0e/gb9N4=
-github.com/projectdiscovery/retryablehttp-go v1.0.58/go.mod h1:bbok7sSEplSwZOY91UlLdVilhavYos1RaCJLJx761V0=
+github.com/projectdiscovery/retryabledns v1.0.85 h1:9aLPWu0bcmtK8bPm/JJyfts28hgWf74UPsSG0KMXrqo=
+github.com/projectdiscovery/retryabledns v1.0.85/go.mod h1:cZe0rydjby+ns2oIY7JmywHvtkwWxPzp3PuQz1rV50E=
+github.com/projectdiscovery/retryablehttp-go v1.0.86 h1:r/rqVrT/fSMe6/syIq1FGd8do/vt+Kgca9pFegyHG88=
+github.com/projectdiscovery/retryablehttp-go v1.0.86/go.mod h1:upk8ItKt9hayUp6Z7E60tH314BAnIUQ5y4KS4x9R90g=
github.com/projectdiscovery/sarif v0.0.1 h1:C2Tyj0SGOKbCLgHrx83vaE6YkzXEVrMXYRGLkKCr/us=
github.com/projectdiscovery/sarif v0.0.1/go.mod h1:cEYlDu8amcPf6b9dSakcz2nNnJsoz4aR6peERwV+wuQ=
github.com/projectdiscovery/stringsutil v0.0.2 h1:uzmw3IVLJSMW1kEg8eCStG/cGbYYZAja8BH3LqqJXMA=
github.com/projectdiscovery/stringsutil v0.0.2/go.mod h1:EJ3w6bC5fBYjVou6ryzodQq37D5c6qbAYQpGmAy+DC0=
-github.com/projectdiscovery/tlsx v1.1.6 h1:iw2zwKbd2+kRQ8J1G4dLmS0CLyemd/tKz1UzcNsC77A=
-github.com/projectdiscovery/tlsx v1.1.6/go.mod h1:s7SRRFdrwIZBK/RXXZi4CR/CubqFSvp8h5Bk1srEZIo=
-github.com/projectdiscovery/uncover v1.0.7 h1:ut+2lTuvmftmveqF5RTjMWAgyLj8ltPQC7siFy9sj0A=
-github.com/projectdiscovery/uncover v1.0.7/go.mod h1:HFXgm1sRPuoN0D4oATljPIdmbo/EEh1wVuxQqo/dwFE=
-github.com/projectdiscovery/useragent v0.0.49 h1:wQc9i+Xy+mUMJ45Ralv1JsQImRWqEOEvpYUe6MchScg=
-github.com/projectdiscovery/useragent v0.0.49/go.mod h1:jQz6X/usiXrPYE6B/1uVKuzIrBJXgw9hLC9eeNy38+0=
-github.com/projectdiscovery/utils v0.0.92 h1:lGCmjUJhzoNX4FQZWpp80058pRlD0/dYxLJOSs07EqY=
-github.com/projectdiscovery/utils v0.0.92/go.mod h1:d5uvD5qcRiK3qxZbBy9eatCqrCSuj9SObL04w/WgXSg=
-github.com/projectdiscovery/wappalyzergo v0.0.120 h1:dphOXnaT3rryo9h9fgbxnAVhtQ1uq61yyQZMYyHz960=
-github.com/projectdiscovery/wappalyzergo v0.0.120/go.mod h1:qW0PP+UBMcdQBBnwk+X6YYFs6huKNvn2BOVs4vQPru0=
+github.com/projectdiscovery/tlsx v1.1.8 h1:Y+VkOp6JmUBb4tci1Fbz9U7ELEQ2irFhm+HS58tHruM=
+github.com/projectdiscovery/tlsx v1.1.8/go.mod h1:6u/dbLuMsLzmux58AWnAB24qh2+Trk0auCK2I9B17Vo=
+github.com/projectdiscovery/uncover v1.0.9 h1:s5RbkD/V4r8QcPkys4gTTqMuRSgXq0JprejqLSopN9Y=
+github.com/projectdiscovery/uncover v1.0.9/go.mod h1:2PUF3SpB5QNIJ8epaB2xbRzkPaxEAWRDm3Ir2ijt81U=
+github.com/projectdiscovery/useragent v0.0.78 h1:YpgiY3qXpzygFA88SWVseAyWeV9ZKrIpDkfOY+mQ/UY=
+github.com/projectdiscovery/useragent v0.0.78/go.mod h1:SQgk2DZu1qCvYqBRYWs2sjenXqLEDnRw65wJJoolwZ4=
+github.com/projectdiscovery/utils v0.2.18 h1:uV5JIYKIq8gXdu9wrCeUq3yqPiSCokTrKuLuZwXMSSw=
+github.com/projectdiscovery/utils v0.2.18/go.mod h1:gcKxBTK1eNF+K8vzD62sMMVFf1eJoTgEiS81mp7CQjI=
+github.com/projectdiscovery/wappalyzergo v0.2.2 h1:AQT6+oo++HOcseTFSTa2en08vWv5miE/NgnJlqL1lCQ=
+github.com/projectdiscovery/wappalyzergo v0.2.2/go.mod h1:k3aujwFsLcB24ppzwNE0lYpV3tednKGJVTbk4JgrhmI=
github.com/projectdiscovery/yamldoc-go v1.0.4 h1:eZoESapnMw6WAHiVgRwNqvbJEfNHEH148uthhFbG5jE=
github.com/projectdiscovery/yamldoc-go v1.0.4/go.mod h1:8PIPRcUD55UbtQdcfFR1hpIGRWG0P7alClXNGt1TBik=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
@@ -927,19 +948,17 @@ github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1
github.com/prometheus/procfs v0.7.3/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
github.com/prometheus/procfs v0.8.0/go.mod h1:z7EfXMXOkbkqb9IINtpCn86r/to3BnA0uaxHdg830/4=
github.com/prometheus/procfs v0.9.0/go.mod h1:+pB4zwohETzFnmlpe6yd2lSc+0/46IYZRB/chUwxUZY=
-github.com/quic-go/quic-go v0.42.0 h1:uSfdap0eveIl8KXnipv9K7nlwZ5IqLlYOpJ58u5utpM=
-github.com/quic-go/quic-go v0.42.0/go.mod h1:132kz4kL3F9vxhW3CtQJLDVwcFe5wdWeJXXijhsO57M=
github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/redis/go-redis/v9 v9.1.0 h1:137FnGdk+EQdCbye1FW+qOEcY5S+SpY9T0NiuqvtfMY=
github.com/redis/go-redis/v9 v9.1.0/go.mod h1:urWj3He21Dj5k4TK1y59xH8Uj6ATueP8AH1cY3lZl4c=
-github.com/refraction-networking/utls v1.6.1 h1:n1JG5karzdGWsI6iZmGrOv3SNzR4c+4M8J6KWGsk3lA=
-github.com/refraction-networking/utls v1.6.1/go.mod h1:+EbcQOvQvXoFV9AEKbuGlljt1doLRKAVY1jJHe9EtDo=
+github.com/refraction-networking/utls v1.6.7 h1:zVJ7sP1dJx/WtVuITug3qYUq034cDq9B2MR1K67ULZM=
+github.com/refraction-networking/utls v1.6.7/go.mod h1:BC3O4vQzye5hqpmDTWUqi4P5DDhzJfkV1tdqtawQIH0=
github.com/remeh/sizedwaitgroup v1.0.0 h1:VNGGFwNo/R5+MJBf6yrsr110p0m4/OX4S3DCy7Kyl5E=
github.com/remeh/sizedwaitgroup v1.0.0/go.mod h1:3j2R4OIe/SeS6YDhICBy22RWjJC5eNCJ1V+9+NVNYlo=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
-github.com/rivo/uniseg v0.4.6 h1:Sovz9sDSwbOz9tgUy8JpT+KgCkPYJEN/oYzlJiYTNLg=
-github.com/rivo/uniseg v0.4.6/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
+github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
+github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
@@ -949,7 +968,6 @@ github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc=
github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
-github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca/go.mod h1:uugorj2VCxiV1x+LzaIdVa9b4S4qGAcH6cbhh4qVxOU=
github.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d h1:hrujxIzL1woJ7AwssoOcM/tq5JjjG2yYOc8odClEiXA=
github.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d/go.mod h1:uugorj2VCxiV1x+LzaIdVa9b4S4qGAcH6cbhh4qVxOU=
github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da/go.mod h1:gi+0XIa01GRL2eRQVjQkKGqKF3SF9vZR/HnPullcV2E=
@@ -970,6 +988,8 @@ github.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFt
github.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ=
github.com/shoenig/test v0.6.4 h1:kVTaSd7WLz5WZ2IaoM0RSzRsUD+m8wRR+5qvntpn4LU=
github.com/shoenig/test v0.6.4/go.mod h1:byHiCGXqrVaflBLAMq/srcZIHynQPQgeyvkvXnjqq0k=
+github.com/shurcooL/graphql v0.0.0-20230722043721-ed46e5a46466 h1:17JxqqJY66GmZVHkmAsGEkcIu0oCe3AM420QDgGwZx0=
+github.com/shurcooL/graphql v0.0.0-20230722043721-ed46e5a46466/go.mod h1:9dIRpgIY7hVhoqfe0/FcYp0bpInZaT7dc3BYOprrIUE=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
@@ -1016,17 +1036,17 @@ github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsT
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE=
github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ=
-github.com/tarunKoyalwar/goleak v0.0.0-20240426214851-746d64600adc h1:/5P5I7oDqdLee8W9Moof0xSD8tT1qEVzhObSI9CqHkg=
-github.com/tarunKoyalwar/goleak v0.0.0-20240426214851-746d64600adc/go.mod h1:uQdBQGrE1fZ2EyOs0pLcCDd1bBV4rSThieuIIGhXZ50=
+github.com/tarunKoyalwar/goleak v0.0.0-20240429141123-0efa90dbdcf9 h1:GXIyLuIJ5Qk46lI8WJ83qHBZKUI3zhmMmuoY9HICUIQ=
+github.com/tarunKoyalwar/goleak v0.0.0-20240429141123-0efa90dbdcf9/go.mod h1:uQdBQGrE1fZ2EyOs0pLcCDd1bBV4rSThieuIIGhXZ50=
github.com/tidwall/assert v0.1.0 h1:aWcKyRBUAdLoVebxo95N7+YZVTFF/ASTr7BN4sLP6XI=
github.com/tidwall/assert v0.1.0/go.mod h1:QLYtGyeqse53vuELQheYl9dngGCJQ+mTtlxcktb+Kj8=
github.com/tidwall/btree v1.7.0 h1:L1fkJH/AuEh5zBnnBbmTwQ5Lt+bRJ5A8EWecslvo9iI=
github.com/tidwall/btree v1.7.0/go.mod h1:twD9XRA5jj9VUQGELzDO4HPQTNJsoWWfYEL+EUQ2cKY=
-github.com/tidwall/buntdb v1.3.0 h1:gdhWO+/YwoB2qZMeAU9JcWWsHSYU3OvcieYgFRS0zwA=
-github.com/tidwall/buntdb v1.3.0/go.mod h1:lZZrZUWzlyDJKlLQ6DKAy53LnG7m5kHyrEHvvcDmBpU=
+github.com/tidwall/buntdb v1.3.1 h1:HKoDF01/aBhl9RjYtbaLnvX9/OuenwvQiC3OP1CcL4o=
+github.com/tidwall/buntdb v1.3.1/go.mod h1:lZZrZUWzlyDJKlLQ6DKAy53LnG7m5kHyrEHvvcDmBpU=
github.com/tidwall/gjson v1.12.1/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
-github.com/tidwall/gjson v1.17.0 h1:/Jocvlh98kcTfpN2+JzGQWQcqrPQwDrVEMApx/M5ZwM=
-github.com/tidwall/gjson v1.17.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
+github.com/tidwall/gjson v1.17.1 h1:wlYEnwqAHgzmhNUFfw7Xalt2JzQvsMx2Se4PcoFCT/U=
+github.com/tidwall/gjson v1.17.1/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/grect v0.1.4 h1:dA3oIgNgWdSspFzn1kS4S/RDpZFLrIxAZOdJKjYapOg=
github.com/tidwall/grect v0.1.4/go.mod h1:9FBsaYRaR0Tcy4UwefBX/UDcDcDy9V5jUcxHzv2jd5Q=
github.com/tidwall/lotsa v1.0.2 h1:dNVBH5MErdaQ/xd9s769R31/n2dXavsQ0Yf4TMEHHw8=
@@ -1057,8 +1077,8 @@ github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4d
github.com/ugorji/go/codec v1.2.11/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg=
github.com/ulikunitz/xz v0.5.8/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/ulikunitz/xz v0.5.9/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
-github.com/ulikunitz/xz v0.5.11 h1:kpFauv27b6ynzBNT/Xy+1k+fK4WswhN/6PN5WhFAGw8=
-github.com/ulikunitz/xz v0.5.11/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
+github.com/ulikunitz/xz v0.5.12 h1:37Nm15o69RwBkXM0J6A5OlE67RZTfzUxTj8fB3dfcsc=
+github.com/ulikunitz/xz v0.5.12/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA=
github.com/urfave/cli v1.22.1/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
@@ -1069,15 +1089,21 @@ github.com/valyala/fasttemplate v1.2.2/go.mod h1:KHLXt3tVN2HBp8eijSv/kGJopbvo7S+
github.com/weppos/publicsuffix-go v0.12.0/go.mod h1:z3LCPQ38eedDQSwmsSRW4Y7t2L8Ln16JPQ02lHAdn5k=
github.com/weppos/publicsuffix-go v0.13.0/go.mod h1:z3LCPQ38eedDQSwmsSRW4Y7t2L8Ln16JPQ02lHAdn5k=
github.com/weppos/publicsuffix-go v0.30.0/go.mod h1:kBi8zwYnR0zrbm8RcuN1o9Fzgpnnn+btVN8uWPMyXAY=
-github.com/weppos/publicsuffix-go v0.30.2-0.20230730094716-a20f9abcc222 h1:h2JizvZl9aIj6za9S5AyrkU+OzIS4CetQthH/ejO+lg=
-github.com/weppos/publicsuffix-go v0.30.2-0.20230730094716-a20f9abcc222/go.mod h1:s41lQh6dIsDWIC1OWh7ChWJXLH0zkJ9KHZVqA7vHyuQ=
+github.com/weppos/publicsuffix-go v0.30.2 h1:Np18yzfMR90jNampWFs7iSh2sw/qCZkhL41/ffyihCU=
+github.com/weppos/publicsuffix-go v0.30.2/go.mod h1:/hGscit36Yt+wammfBBwdMdxBT8btsTt6KvwO9OvMyM=
github.com/weppos/publicsuffix-go/publicsuffix/generator v0.0.0-20220927085643-dc0d00c92642/go.mod h1:GHfoeIdZLdZmLjMlzBftbTDntahTttUMWjxZwQJhULE=
github.com/wk8/go-ordered-map/v2 v2.1.8 h1:5h/BUHu93oj4gIdvHHHGsScSTMijfx5PeYkE/fJgbpc=
github.com/wk8/go-ordered-map/v2 v2.1.8/go.mod h1:5nJHM5DyteebpVlHnWMV0rPz6Zp7+xBAnxjb1X5vnTw=
-github.com/xanzy/go-gitlab v0.84.0 h1:PdpCaskQSgcVDsx21c6ikf8Rfyo7SNtFAJwP9PrbCFE=
-github.com/xanzy/go-gitlab v0.84.0/go.mod h1:5ryv+MnpZStBH8I/77HuQBsMbBGANtVpLWC15qOjWAw=
+github.com/xanzy/go-gitlab v0.107.0 h1:P2CT9Uy9yN9lJo3FLxpMZ4xj6uWcpnigXsjvqJ6nd2Y=
+github.com/xanzy/go-gitlab v0.107.0/go.mod h1:wKNKh3GkYDMOsGmnfuX+ITCmDuSDWFO0G+C4AygL9RY=
github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM=
github.com/xanzy/ssh-agent v0.3.3/go.mod h1:6dzNDKs0J9rVPHPhaGCukekBHKqfl+L3KghI1Bc68Uw=
+github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
+github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
+github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
+github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
+github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
+github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb h1:zGWFAtiMcyryUHoUjUJX0/lt1H2+i2Ka2n+D3DImSNo=
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
@@ -1089,8 +1115,12 @@ github.com/xhit/go-str2duration v1.2.0/go.mod h1:3cPSlfZlUHVlneIVfePFWcJZsuwf+P1
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofmx9yWTog9BfvIu0q41lo=
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8/go.mod h1:HUYIGzjTL3rfEspMxjDjgmT5uz5wzYJKVo23qUhYTos=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
+github.com/yassinebenaid/godump v0.10.0 h1:FolBA+Ix5uwUiXYBBYOsf1VkT5+0f4gtFNTkYTiIR08=
+github.com/yassinebenaid/godump v0.10.0/go.mod h1:dc/0w8wmg6kVIvNGAzbKH1Oa54dXQx8SNKh4dPRyW44=
github.com/yl2chen/cidranger v1.0.2 h1:lbOWZVCG1tCRX4u24kuM1Tb4nHqWkDxwLdoS+SevawU=
github.com/yl2chen/cidranger v1.0.2/go.mod h1:9U1yz7WPYDwf0vpNWFaeRh0bjwz5RVgRy/9UEQfHl0g=
+github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 h1:ilQV1hzziu+LLM3zUTJ0trRztfwgjqKnBWNtSRkbmwM=
+github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfSfmXjznFBSZNN13rSJjlIOI1fUNAtF7rmI=
github.com/ysmood/fetchup v0.2.3 h1:ulX+SonA0Vma5zUFXtv52Kzip/xe7aj4vqT5AJwQ+ZQ=
github.com/ysmood/fetchup v0.2.3/go.mod h1:xhibcRKziSvol0H1/pj33dnKrYyI2ebIvz5cOOkYGns=
github.com/ysmood/goob v0.4.0 h1:HsxXhyLBeGzWXnqVKtmT9qM7EuVs/XOgkX7T6r1o1AQ=
@@ -1110,11 +1140,11 @@ github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9de
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
-github.com/yuin/goldmark v1.5.2/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
-github.com/yuin/goldmark v1.5.4 h1:2uY/xC0roWy8IBEGLgB1ywIoEJFGmRrX21YQcvGZzjU=
-github.com/yuin/goldmark v1.5.4/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
-github.com/yuin/goldmark-emoji v1.0.1 h1:ctuWEyzGBwiucEqxzwe0SOYDXPAucOrE9NQC18Wa1os=
-github.com/yuin/goldmark-emoji v1.0.1/go.mod h1:2w1E6FEWLcDQkoTE+7HU6QF1F6SLlNGjRIBbIZQFqkQ=
+github.com/yuin/goldmark v1.7.1/go.mod h1:uzxRWxtg69N339t3louHJ7+O03ezfj6PlliRlaOzY1E=
+github.com/yuin/goldmark v1.7.4 h1:BDXOHExt+A7gwPCJgPIIq7ENvceR7we7rOS9TNoLZeg=
+github.com/yuin/goldmark v1.7.4/go.mod h1:uzxRWxtg69N339t3louHJ7+O03ezfj6PlliRlaOzY1E=
+github.com/yuin/goldmark-emoji v1.0.3 h1:aLRkLHOuBR2czCY4R8olwMjID+tENfhyFDMCRhbIQY4=
+github.com/yuin/goldmark-emoji v1.0.3/go.mod h1:tTkZEbwu5wkPmgTcitqddVxY9osFZiavD+r4AzQrh1U=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
github.com/zcalusic/sysinfo v1.0.2 h1:nwTTo2a+WQ0NXwo0BGRojOJvJ/5XKvQih+2RrtWqfxc=
@@ -1133,16 +1163,18 @@ github.com/zmap/zcertificate v0.0.1/go.mod h1:q0dlN54Jm4NVSSuzisusQY0hqDWvu92C+T
github.com/zmap/zcrypto v0.0.0-20201128221613-3719af1573cf/go.mod h1:aPM7r+JOkfL+9qSB4KbYjtoEzJqUK50EXkkJabeNJDQ=
github.com/zmap/zcrypto v0.0.0-20201211161100-e54a5822fb7e/go.mod h1:aPM7r+JOkfL+9qSB4KbYjtoEzJqUK50EXkkJabeNJDQ=
github.com/zmap/zcrypto v0.0.0-20230310154051-c8b263fd8300/go.mod h1:mOd4yUMgn2fe2nV9KXsa9AyQBFZGzygVPovsZR+Rl5w=
-github.com/zmap/zcrypto v0.0.0-20231219022726-a1f61fb1661c h1:U1b4THKcgOpJ+kILupuznNwPiURtwVW3e9alJvji9+s=
-github.com/zmap/zcrypto v0.0.0-20231219022726-a1f61fb1661c/go.mod h1:GSDpFDD4TASObxvfZfvpZZ3OWHIUHMlhVWlkOe4ewVk=
+github.com/zmap/zcrypto v0.0.0-20240512203510-0fef58d9a9db h1:IfONOhyZlf4qPt3ENPU+27mBbPjzTQ+swKpj7MJva9I=
+github.com/zmap/zcrypto v0.0.0-20240512203510-0fef58d9a9db/go.mod h1:mo/07mo6reDaiz6BzveCuYBWb1d+aX8Pf8Nh+Q57y2g=
github.com/zmap/zflags v1.4.0-beta.1.0.20200204220219-9d95409821b6/go.mod h1:HXDUD+uue8yeLHr0eXx1lvY6CvMiHbTKw5nGmA9OUoo=
github.com/zmap/zgrab2 v0.1.8-0.20230806160807-97ba87c0e706 h1:LaMyYFWQA7kh3ovPfAaFDTKlJu3JGng8khruOtsBVnE=
github.com/zmap/zgrab2 v0.1.8-0.20230806160807-97ba87c0e706/go.mod h1:re2kMcs84XHb8Xl6RInt0emoKCuphfmfjHYuteviLHQ=
github.com/zmap/zlint/v3 v3.0.0/go.mod h1:paGwFySdHIBEMJ61YjoqT4h7Ge+fdYG4sUQhnTb1lJ8=
go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
-go.etcd.io/bbolt v1.3.8 h1:xs88BrvEv273UsB79e0hcVrlUWmS0a8upikMFhSyAtA=
-go.etcd.io/bbolt v1.3.8/go.mod h1:N9Mkw9X8x5fupy0IKsmuqVtoGDyxsaDlbk4Rd05IAQw=
+go.etcd.io/bbolt v1.3.10 h1:+BqfJTcCzTItrop8mq/lbzL8wSGtj94UO/3U31shqG0=
+go.etcd.io/bbolt v1.3.10/go.mod h1:bK3UQLPJZly7IlNmV7uVHJDxfe5aK9Ll93e/74Y9oEQ=
go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738/go.mod h1:dnLIgRNXwCJa5e+c6mIZCrds/GIG4ncV9HhK5PX7jPg=
+go.mongodb.org/mongo-driver v1.17.0 h1:Hp4q2MCjvY19ViwimTs00wHi7G4yzxh4/2+nTx8r40k=
+go.mongodb.org/mongo-driver v1.17.0/go.mod h1:wwWm/+BuOddhcq3n68LKRmgk2wXzmF6s0SFOa0GINL4=
go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
go.opencensus.io v0.20.2/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
@@ -1178,7 +1210,6 @@ golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200510223506-06a226fb4e37/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/crypto v0.0.0-20201016220609-9e8e0b390897/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20201124201722-c8d3bf9c5392/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20201208171446-5f87f3452ae9/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
@@ -1189,10 +1220,11 @@ golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e/go.mod h1:IxCIyHEi3zRg3s0
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
-golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio=
-golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4=
-golang.org/x/crypto v0.22.0 h1:g1v0xeRhjcugydODzvb3mEM9SQ0HGp9s/nh3COQ/C30=
-golang.org/x/crypto v0.22.0/go.mod h1:vr6Su+7cTlO45qkww3VDJlzDn0ctJvRgYbC2NvXHt+M=
+golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
+golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
+golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
+golang.org/x/crypto v0.28.0 h1:GBDwsMXVQi34v5CCYUm2jkJvu4cbtru2U4TN2PSyQnw=
+golang.org/x/crypto v0.28.0/go.mod h1:rmgy+3RHxRZMyY0jjAJShp2zgEdOqj2AO7U0pYmeQ7U=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -1203,8 +1235,8 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
-golang.org/x/exp v0.0.0-20240119083558-1b970713d09a h1:Q8/wZp0KX97QFTc2ywcOE0YRjZPVIx+MXInMzdvQqcA=
-golang.org/x/exp v0.0.0-20240119083558-1b970713d09a/go.mod h1:idGWGoKP1toJGkd5/ig9ZLuPcZBC3ewk7SzmH0uou08=
+golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 h1:vr/HnozRka3pE4EsMEg1lgkXJkTFJCVUX+S/ZT6wYzM=
+golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842/go.mod h1:XtvwrStGgqGPLc4cjQfWqZHG1YFdYs6swckp8vpsjnc=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@@ -1227,8 +1259,8 @@ golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
-golang.org/x/mod v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0=
-golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
+golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA=
+golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -1271,22 +1303,22 @@ golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwY
golang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
-golang.org/x/net v0.0.0-20210610132358-84b48f89b13b/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
-golang.org/x/net v0.0.0-20221002022538-bcab6841153b/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk=
golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
-golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA=
-golang.org/x/net v0.24.0 h1:1PcaxkF854Fu3+lvBIx5SYn9wRlBzzcnHZSiaFFAb0w=
-golang.org/x/net v0.24.0/go.mod h1:2Q7sJY5mzlzWjKtYUEXSlBWCdyaioyXzRB2RtU8KVE8=
+golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
+golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
+golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
+golang.org/x/net v0.30.0 h1:AcW1SDZMkb8IpzCdQUaIq2sP4sZ4zw+55h6ynffypl4=
+golang.org/x/net v0.30.0/go.mod h1:2wGyMJ5iFasEhkwi13ChkO/t1ECNC4X4eBKkVFyYFlU=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1296,8 +1328,8 @@ golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ
golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
golang.org/x/oauth2 v0.5.0/go.mod h1:9/XBHVqLaWO3/BRHs5jbpYCnOZVjj5V0ndyaAM7KB4I=
golang.org/x/oauth2 v0.6.0/go.mod h1:ycmewcwgD4Rpr3eZJLSB4Kyyljb3qDh40vJ8STE5HKw=
-golang.org/x/oauth2 v0.11.0 h1:vPL4xzxBM4niKCW6g9whtaWVXTJf1U5e4aZxxFx/gbU=
-golang.org/x/oauth2 v0.11.0/go.mod h1:LdF7O/8bLR/qWK9DrpXmbHLTouvRHK0SgJl0GmDBchk=
+golang.org/x/oauth2 v0.22.0 h1:BzDx2FehcG7jJwgWLELCdmLuxk2i+x9UDpSiss2u0ZA=
+golang.org/x/oauth2 v0.22.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -1311,8 +1343,8 @@ golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ=
-golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
+golang.org/x/sync v0.8.0 h1:3NFvSEYkUoMifnESzZl15y791HH1qU2xm6eCJU5ZPXQ=
+golang.org/x/sync v0.8.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -1363,7 +1395,6 @@ golang.org/x/sys v0.0.0-20210228012217-479acdf4ea46/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20210616045830-e2b7044e8c71/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -1376,20 +1407,20 @@ golang.org/x/sys v0.0.0-20220330033206-e17cdc41300f/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.19.0 h1:q5f1RH2jigJ1MoAWp2KTp3gm5zAGFUTarQZ5U386+4o=
-golang.org/x/sys v0.19.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo=
+golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -1399,10 +1430,11 @@ golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
-golang.org/x/term v0.10.0/go.mod h1:lpqdcUyK/oCiQxvxVrppt5ggO2KCZ5QblwqPnfZ6d5o=
-golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0=
-golang.org/x/term v0.19.0 h1:+ThwsDv+tYfnJFhF4L8jITxu1tdTWRTZpdsWgEgjL6Q=
-golang.org/x/term v0.19.0/go.mod h1:2CuTdWZ7KHSQwUzKva0cbMg6q2DMI3Mmxp+gKJbskEk=
+golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U=
+golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
+golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
+golang.org/x/term v0.25.0 h1:WtHI/ltw4NvSUig5KARz9h521QvRC8RmF/cuYqifU24=
+golang.org/x/term v0.25.0/go.mod h1:RPyXicDX+6vLxogjjRxjgD2TKtmAO6NZBsBRfrOLu7M=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -1416,15 +1448,17 @@ golang.org/x/text v0.6.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
-golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
-golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
+golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
+golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
+golang.org/x/text v0.19.0 h1:kTxAhCbGbxhK0IwgSKiMO5awPoDQ0RpfiVYBfK860YM=
+golang.org/x/text v0.19.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
-golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk=
-golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
+golang.org/x/time v0.6.0 h1:eTDhh4ZXt5Qf0augr54TN6suAUudPcawVZeIAPU7D4U=
+golang.org/x/time v0.6.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -1476,8 +1510,8 @@ golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
-golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc=
-golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps=
+golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg=
+golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -1506,7 +1540,6 @@ google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
-google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c=
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
@@ -1570,8 +1603,9 @@ google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQ
google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
-google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI=
google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
+google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
+google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -1605,7 +1639,6 @@ gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools/v3 v3.3.0 h1:MfDY1b1/0xN1CyMlQDac0ziEy9zJQd9CXBRRDHw2jJo=
diff --git a/integration_tests/flow/conditional-flow.yaml b/integration_tests/flow/conditional-flow.yaml
index dd76c89b54..8496016acc 100644
--- a/integration_tests/flow/conditional-flow.yaml
+++ b/integration_tests/flow/conditional-flow.yaml
@@ -14,7 +14,7 @@ dns:
matchers:
- type: word
words:
- - "ghost.io"
+ - ".vercel-dns.com"
internal: true
http:
@@ -25,4 +25,4 @@ http:
matchers:
- type: word
words:
- - "ghost.io"
\ No newline at end of file
+ - "html>"
\ No newline at end of file
diff --git a/integration_tests/fuzz/fuzz-path-sqli.yaml b/integration_tests/fuzz/fuzz-path-sqli.yaml
index e23098d931..531427becb 100644
--- a/integration_tests/fuzz/fuzz-path-sqli.yaml
+++ b/integration_tests/fuzz/fuzz-path-sqli.yaml
@@ -15,21 +15,18 @@ http:
- type: dsl
dsl:
- 'method == "GET"'
- - regex("/(.*?/)([0-9]+)(/.*)?",path)
condition: and
payloads:
pathsqli:
- - "'OR1=1"
- '%20OR%20True'
fuzzing:
- part: path
- type: replace-regex
+ type: postfix
mode: single
- replace-regex: '/(.*?/)([0-9]+)(/.*)?'
fuzz:
- - '/${1}${2}{{pathsqli}}${3}'
+ - '{{pathsqli}}'
matchers:
- type: status
diff --git a/integration_tests/protocols/code/pre-condition.yaml b/integration_tests/protocols/code/pre-condition.yaml
index 1c44e9579a..be9f1c908c 100644
--- a/integration_tests/protocols/code/pre-condition.yaml
+++ b/integration_tests/protocols/code/pre-condition.yaml
@@ -23,4 +23,4 @@ code:
- type: dsl
dsl:
- true
-# digest: 490a004630440220192fb8f704b078c2885047b85ac1a0491be86485c033a976d201599683a35aab0220604b1c3781e9d97079d0e5c23c18e6a2d87493c8e2b930536e692ee7d06e9247:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
+# digest: 490a00463044022048c083c338c0195f5012122d40c1009d2e2030c583e56558e0d6249a41e6f3f4022070656adf748f4874018d7a01fce116db10a3acd1f9b03e12a83906fb625b5c50:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
diff --git a/integration_tests/protocols/code/py-env-var.yaml b/integration_tests/protocols/code/py-env-var.yaml
index 9ff947c208..c37759954c 100644
--- a/integration_tests/protocols/code/py-env-var.yaml
+++ b/integration_tests/protocols/code/py-env-var.yaml
@@ -20,4 +20,4 @@ code:
- type: word
words:
- "hello from input baz"
-# digest: 4a0a00473045022033f72f1b9d5143f58a2dc79c2597000f34080251ac3702c36c3fad00917dfeeb0221009ba05c715c9e2e36dba471be6c0106a09ae3822d8a3e9e4bcf377e9f4a395a01:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
+# digest: 4b0a00483046022100cbbdb7214f669d111b671d271110872dc8af2ab41cf5c312b6e4f64126f55337022100a60547952a0c2bea58388f2d2effe8ad73cd6b6fc92e73eb3c8f88beab6105ec:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
diff --git a/integration_tests/protocols/code/py-file.yaml b/integration_tests/protocols/code/py-file.yaml
index ad69371d05..3c7e00bca8 100644
--- a/integration_tests/protocols/code/py-file.yaml
+++ b/integration_tests/protocols/code/py-file.yaml
@@ -18,4 +18,4 @@ code:
- type: word
words:
- "hello from input"
-# digest: 4a0a004730450220377128cb11d9f6f0fee1f4dbd841e46783de26e90a216fa55a7609ee2bc823c60221009166ee0f85e3a1811588ab19e73ea96ab3d582dc8180dbcbbad0ea9ab7e9025d:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
+# digest: 4a0a00473045022032b81e8bb7475abf27639b0ced71355497166d664698021f26498e7031d62a23022100e99ccde578bfc0b658f16427ae9a3d18922849d3ba3e022032ea0d2a8e77fadb:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
diff --git a/integration_tests/protocols/code/py-interactsh.yaml b/integration_tests/protocols/code/py-interactsh.yaml
index 76d14efb2e..fdf7a5d632 100644
--- a/integration_tests/protocols/code/py-interactsh.yaml
+++ b/integration_tests/protocols/code/py-interactsh.yaml
@@ -26,4 +26,4 @@ code:
part: interactsh_protocol
words:
- "http"
-# digest: 4b0a00483046022100d472d50bd83117d334f5217c7a40dcdf34138e90029eaace51697d902296bf37022100a393b49420a96f60d6d89b79b5135ee2233b2468d374851890eea114b08195d1:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
+# digest: 4a0a0047304502201a5dd0eddfab4f02588a5a8ac1947a5fa41fed80b59d698ad5cc00456296efb6022100fe6e608e38c060964800f5f863a7cdc93f686f2d0f4b52854f73948b808b4511:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
diff --git a/integration_tests/protocols/code/py-snippet.yaml b/integration_tests/protocols/code/py-snippet.yaml
index 4837fa7e0e..0762f58bf7 100644
--- a/integration_tests/protocols/code/py-snippet.yaml
+++ b/integration_tests/protocols/code/py-snippet.yaml
@@ -21,4 +21,4 @@ code:
- type: word
words:
- "hello from input"
-# digest: 4b0a004830460221008886054bb5dd6345e434e30f31c8fddce3c484a4f33aa6321b5185675866029d022100d188a83d0fde029f8b586061c65ab72b43755c3fb10fdd59501bb9bbadbb1ff7:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
+# digest: 4b0a00483046022100ced1702728cc68f906c4c7d2c4d05ed071bfabee1e36eec7ebecbeca795a170c022100d20fd41796f130a8f9c4972fee85386d67d61eb5fc1119b1afe2a851eb2f3e65:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
diff --git a/integration_tests/protocols/multi/dynamic-values.yaml b/integration_tests/protocols/multi/dynamic-values.yaml
index 2bd3113488..4872c6bf3d 100644
--- a/integration_tests/protocols/multi/dynamic-values.yaml
+++ b/integration_tests/protocols/multi/dynamic-values.yaml
@@ -13,7 +13,7 @@ dns:
- type: dsl
name: blogid
dsl:
- - trim_suffix(cname,'.ghost.io')
+ - trim_suffix(cname,'.vercel-dns.com')
internal: true
http:
@@ -24,6 +24,6 @@ http:
matchers:
- type: dsl
dsl:
- - contains(body,'ProjectDiscovery.io') # check for http string
- - blogid == 'projectdiscovery' # check for cname (extracted information from dns response)
+ - contains(body,'introduction') # check for http string
+ - blogid == 'cname' # check for cname (extracted information from dns response)
condition: and
\ No newline at end of file
diff --git a/integration_tests/protocols/multi/evaluate-variables.yaml b/integration_tests/protocols/multi/evaluate-variables.yaml
index f1a6fd98c0..7d0338ca23 100644
--- a/integration_tests/protocols/multi/evaluate-variables.yaml
+++ b/integration_tests/protocols/multi/evaluate-variables.yaml
@@ -7,7 +7,7 @@ info:
variables:
- cname_filtered: '{{trim_suffix(dns_cname,".ghost.io")}}'
+ cname_filtered: '{{trim_suffix(dns_cname,".vercel-dns.com")}}'
dns:
- name: "{{FQDN}}" # DNS Request
@@ -24,7 +24,7 @@ http:
matchers:
- type: dsl
dsl:
- - contains(http_body,'ProjectDiscovery.io') # check for http string
- - cname_filtered == 'projectdiscovery' # check for cname (extracted information from dns response)
- - ssl_subject_cn == 'blog.projectdiscovery.io'
+ - contains(http_body,'introduction') # check for http string
+ - cname_filtered == 'cname' # check for cname (extracted information from dns response)
+ - ssl_subject_cn == 'docs.projectdiscovery.io'
condition: and
\ No newline at end of file
diff --git a/integration_tests/protocols/multi/exported-response-vars.yaml b/integration_tests/protocols/multi/exported-response-vars.yaml
index 1edfa65f34..b6b9417232 100644
--- a/integration_tests/protocols/multi/exported-response-vars.yaml
+++ b/integration_tests/protocols/multi/exported-response-vars.yaml
@@ -20,7 +20,7 @@ http:
matchers:
- type: dsl
dsl:
- - contains(http_body,'ProjectDiscovery.io') # check for http string
- - trim_suffix(dns_cname,'.ghost.io') == 'projectdiscovery' # check for cname (extracted information from dns response)
- - ssl_subject_cn == 'blog.projectdiscovery.io'
+ - contains(http_body,'introduction') # check for http string
+ - trim_suffix(dns_cname,'.vercel-dns.com') == 'cname' # check for cname (extracted information from dns response)
+ - ssl_subject_cn == 'docs.projectdiscovery.io'
condition: and
\ No newline at end of file
diff --git a/integration_tests/protocols/ssl/multi-req.yaml b/integration_tests/protocols/ssl/multi-req.yaml
new file mode 100644
index 0000000000..670ca633d9
--- /dev/null
+++ b/integration_tests/protocols/ssl/multi-req.yaml
@@ -0,0 +1,34 @@
+id: multi-req
+
+info:
+ name: Multi-Request
+ author: pdteam
+ severity: info
+
+ssl:
+ - address: "{{Host}}:{{Port}}"
+ min_version: ssl30
+ max_version: ssl30
+
+ extractors:
+ - type: json
+ json:
+ - " .tls_version"
+
+ - address: "{{Host}}:{{Port}}"
+ min_version: tls10
+ max_version: tls10
+
+ extractors:
+ - type: json
+ json:
+ - " .tls_version"
+
+ - address: "{{Host}}:{{Port}}"
+ min_version: tls11
+ max_version: tls11
+
+ extractors:
+ - type: json
+ json:
+ - " .tls_version"
\ No newline at end of file
diff --git a/integration_tests/workflow/code-template-1.yaml b/integration_tests/workflow/code-template-1.yaml
index d41a1a6957..1d8f9154fe 100644
--- a/integration_tests/workflow/code-template-1.yaml
+++ b/integration_tests/workflow/code-template-1.yaml
@@ -19,4 +19,4 @@ code:
regex:
- 'hello from (.*)'
group: 1
-# digest: 490a0046304402202c63d47bb0acdd40b3b852d95490d492ff5741b84071b2a8a40371be7797c13602202b6b977e157edf2ef70a402a2e57d4eb5a67c5ca91f0a2f9a10a966e8485ebaf:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
+# digest: 490a0046304402206b3648e8d393ac4df82c7d59b1a6ee3731c66c249dbd4d9bf31f0b7f176b37ec02203184d36373e516757c7d708b5799bc16edb1cebc0a64f3442d13ded4b33c42fb:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
diff --git a/integration_tests/workflow/code-template-2.yaml b/integration_tests/workflow/code-template-2.yaml
index ddd7d36a4e..c135b52251 100644
--- a/integration_tests/workflow/code-template-2.yaml
+++ b/integration_tests/workflow/code-template-2.yaml
@@ -18,4 +18,4 @@ code:
- type: word
words:
- "hello from first"
-# digest: 490a00463044022025661eab353b7f359c0d428a86b6287545d7f759375e8025cc8c9c77b616ca6502200bc2c019059622df3c88e7caa6dd7d1fb9b956010aa0de2ee2b9f7dd0a3c4954:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
+# digest: 490a0046304402204cbb1bdf8370e49bb930b17460fb35e15f285a3b48b165736ac0e7ba2f9bc0fb022067c134790c4a2cf646b195aa4488e2c222266436e6bda47931908a28807bdb81:4a3eb6b4988d95847d4203be25ed1d46
\ No newline at end of file
diff --git a/integration_tests/workflow/multimatch-value-share-template.yaml b/integration_tests/workflow/multimatch-value-share-template.yaml
new file mode 100644
index 0000000000..fd66975e26
--- /dev/null
+++ b/integration_tests/workflow/multimatch-value-share-template.yaml
@@ -0,0 +1,23 @@
+id: multimatch-value-share-template
+
+info:
+ name: MultiMatch Value Share Template
+ author: tovask
+ severity: info
+
+http:
+ - path:
+ - "{{BaseURL}}/path1?v=1"
+ - "{{BaseURL}}/path1?v=2"
+ matchers:
+ - type: word
+ name: test-matcher
+ words:
+ - "href"
+ extractors:
+ - type: regex
+ part: body
+ name: extracted
+ regex:
+ - 'href="(.*)"'
+ group: 1
diff --git a/integration_tests/workflow/multimatch-value-share-workflow.yaml b/integration_tests/workflow/multimatch-value-share-workflow.yaml
new file mode 100644
index 0000000000..f197f28864
--- /dev/null
+++ b/integration_tests/workflow/multimatch-value-share-workflow.yaml
@@ -0,0 +1,21 @@
+id: multimatch-value-share-workflow
+
+info:
+ name: MultiMatch Value Share Workflow
+ author: tovask
+ severity: info
+ description: Workflow to test value sharing when multiple matches occur in the extractor template
+
+workflows:
+ - template: workflow/multimatch-value-share-template.yaml
+ subtemplates:
+ - template: workflow/match-1.yaml
+ subtemplates:
+ - template: workflow/http-value-share-template-2.yaml
+ - template: workflow/multimatch-value-share-template.yaml
+ matchers:
+ - name: test-matcher
+ subtemplates:
+ - template: workflow/match-1.yaml
+ subtemplates:
+ - template: workflow/http-value-share-template-2.yaml
diff --git a/internal/colorizer/colorizer.go b/internal/colorizer/colorizer.go
index 40f7a80b44..a56f274745 100644
--- a/internal/colorizer/colorizer.go
+++ b/internal/colorizer/colorizer.go
@@ -5,7 +5,15 @@ import (
"github.com/logrusorgru/aurora"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
+=======
+<<<<<<< HEAD:v2/internal/colorizer/colorizer.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:internal/colorizer/colorizer.go
+>>>>>>> projectdiscovery-main
)
const (
diff --git a/internal/pdcp/utils.go b/internal/pdcp/utils.go
index 5d4fa4e1f0..3385d5cad5 100644
--- a/internal/pdcp/utils.go
+++ b/internal/pdcp/utils.go
@@ -5,9 +5,17 @@ import (
urlutil "github.com/projectdiscovery/utils/url"
)
-func getScanDashBoardURL(id string) string {
+func getScanDashBoardURL(id string, teamID string) string {
ux, _ := urlutil.Parse(pdcpauth.DashBoardURL)
ux.Path = "/scans/" + id
+ if ux.Params == nil {
+ ux.Params = urlutil.NewOrderedParams()
+ }
+ if teamID != "" {
+ ux.Params.Add("team_id", teamID)
+ } else {
+ ux.Params.Add("team_id", NoneTeamID)
+ }
ux.Update()
return ux.String()
}
diff --git a/internal/pdcp/writer.go b/internal/pdcp/writer.go
index e6a1da277f..b85fc5121b 100644
--- a/internal/pdcp/writer.go
+++ b/internal/pdcp/writer.go
@@ -18,7 +18,9 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/retryablehttp-go"
pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
+ "github.com/projectdiscovery/utils/env"
errorutil "github.com/projectdiscovery/utils/errors"
+ unitutils "github.com/projectdiscovery/utils/unit"
updateutils "github.com/projectdiscovery/utils/update"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -26,14 +28,18 @@ import (
const (
uploadEndpoint = "/v1/scans/import"
appendEndpoint = "/v1/scans/%s/import"
- flushTimer = time.Duration(1) * time.Minute
- MaxChunkSize = 1024 * 1024 * 4 // 4 MB
+ flushTimer = time.Minute
+ MaxChunkSize = 4 * unitutils.Mega // 4 MB
xidRe = `^[a-z0-9]{20}$`
+ teamIDHeader = "X-Team-Id"
+ NoneTeamID = "none"
)
var (
xidRegex = regexp.MustCompile(xidRe)
_ output.Writer = &UploadWriter{}
+ // teamID if given
+ TeamIDEnv = env.GetEnvOrDefault("PDCP_TEAM_ID", NoneTeamID)
)
// UploadWriter is a writer that uploads its output to pdcp
@@ -48,6 +54,7 @@ type UploadWriter struct {
scanID string
scanName string
counter atomic.Int32
+ TeamID string
}
// NewUploadWriter creates a new upload writer
@@ -56,8 +63,9 @@ func NewUploadWriter(ctx context.Context, creds *pdcpauth.PDCPCredentials) (*Upl
return nil, fmt.Errorf("no credentials provided")
}
u := &UploadWriter{
- creds: creds,
- done: make(chan struct{}, 1),
+ creds: creds,
+ done: make(chan struct{}, 1),
+ TeamID: NoneTeamID,
}
var err error
reader, writer := io.Pipe()
@@ -105,6 +113,14 @@ func (u *UploadWriter) SetScanName(name string) {
u.scanName = name
}
+func (u *UploadWriter) SetTeamID(id string) {
+ if id == "" {
+ u.TeamID = NoneTeamID
+ } else {
+ u.TeamID = id
+ }
+}
+
func (u *UploadWriter) autoCommit(ctx context.Context, r *io.PipeReader) {
reader := bufio.NewReader(r)
ch := make(chan string, 4)
@@ -131,13 +147,13 @@ func (u *UploadWriter) autoCommit(ctx context.Context, r *io.PipeReader) {
if u.scanID == "" {
gologger.Verbose().Msgf("Scan results upload to cloud skipped, no results found to upload")
} else {
- gologger.Info().Msgf("%v Scan results uploaded to cloud, you can view scan results at %v", u.counter.Load(), getScanDashBoardURL(u.scanID))
+ gologger.Info().Msgf("%v Scan results uploaded to cloud, you can view scan results at %v", u.counter.Load(), getScanDashBoardURL(u.scanID, u.TeamID))
}
}()
// temporary buffer to store the results
buff := &bytes.Buffer{}
ticker := time.NewTicker(flushTimer)
-
+ defer ticker.Stop()
for {
select {
case <-ctx.Done():
@@ -184,7 +200,7 @@ func (u *UploadWriter) uploadChunk(buff *bytes.Buffer) error {
// if successful, reset the buffer
buff.Reset()
// log in verbose mode
- gologger.Warning().Msgf("Uploaded results chunk, you can view scan results at %v", getScanDashBoardURL(u.scanID))
+ gologger.Warning().Msgf("Uploaded results chunk, you can view scan results at %v", getScanDashBoardURL(u.scanID, u.TeamID))
return nil
}
@@ -243,6 +259,9 @@ func (u *UploadWriter) getRequest(bin []byte) (*retryablehttp.Request, error) {
req.URL.Update()
req.Header.Set(pdcpauth.ApiKeyHeaderName, u.creds.APIKey)
+ if u.TeamID != NoneTeamID && u.TeamID != "" {
+ req.Header.Set(teamIDHeader, u.TeamID)
+ }
req.Header.Set("Content-Type", "application/octet-stream")
req.Header.Set("Accept", "application/json")
return req, nil
diff --git a/internal/runner/banner.go b/internal/runner/banner.go
index d5a67a55b0..0c5bbbe548 100644
--- a/internal/runner/banner.go
+++ b/internal/runner/banner.go
@@ -6,7 +6,16 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
+=======
+<<<<<<< HEAD:v2/internal/runner/banner.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:internal/runner/banner.go
+>>>>>>> projectdiscovery-main
updateutils "github.com/projectdiscovery/utils/update"
)
diff --git a/internal/runner/healthcheck.go b/internal/runner/healthcheck.go
index 72385853b5..fe4676b15c 100644
--- a/internal/runner/healthcheck.go
+++ b/internal/runner/healthcheck.go
@@ -6,8 +6,18 @@ import (
"runtime"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/internal/runner/healthcheck.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:internal/runner/healthcheck.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
)
diff --git a/internal/runner/lazy.go b/internal/runner/lazy.go
index 01eca98147..1f797d2a25 100644
--- a/internal/runner/lazy.go
+++ b/internal/runner/lazy.go
@@ -3,7 +3,9 @@ package runner
import (
"context"
"fmt"
+ "strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/authprovider/authx"
"github.com/Explorer1092/nuclei/v3/pkg/catalog"
"github.com/Explorer1092/nuclei/v3/pkg/catalog/loader"
@@ -13,6 +15,20 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/helpers/writer"
"github.com/Explorer1092/nuclei/v3/pkg/scan"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/authprovider/authx"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/writer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/replacer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/utils/env"
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
)
@@ -51,6 +67,7 @@ func GetAuthTmplStore(opts types.Options, catalog catalog.Catalog, execOpts prot
opts.ExcludeProtocols = nil
opts.IncludeConditions = nil
cfg := loader.NewConfig(&opts, catalog, execOpts)
+ cfg.StoreId = loader.AuthStoreId
store, err := loader.New(cfg)
if err != nil {
return nil, errorutil.NewWithErr(err).Msgf("failed to initialize dynamic auth templates store")
@@ -74,7 +91,25 @@ func GetLazyAuthFetchCallback(opts *AuthLazyFetchOptions) authx.LazyFetchSecret
vars := map[string]interface{}{}
mainCtx := context.Background()
ctx := scan.NewScanContext(mainCtx, contextargs.NewWithInput(mainCtx, d.Input))
+
+ cliVars := map[string]interface{}{}
+ if opts.ExecOpts.Options != nil {
+ // gets variables passed from cli -v and -env-vars
+ cliVars = generators.BuildPayloadFromOptions(opts.ExecOpts.Options)
+ }
+
for _, v := range d.Variables {
+ // Check if the template has any env variables and expand them
+ if strings.HasPrefix(v.Value, "$") {
+ env.ExpandWithEnv(&v.Value)
+ }
+ if strings.Contains(v.Value, "{{") {
+ // if variables had value like {{username}}, then replace it with the value from cliVars
+ // variables:
+ // - key: username
+ // value: {{username}}
+ v.Value = replacer.Replace(v.Value, cliVars)
+ }
vars[v.Key] = v.Value
ctx.Input.Add(v.Key, v.Value)
}
diff --git a/internal/runner/options.go b/internal/runner/options.go
index 956a6ec5cc..516e4f6f84 100644
--- a/internal/runner/options.go
+++ b/internal/runner/options.go
@@ -29,6 +29,29 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/gologger/formatter"
"github.com/projectdiscovery/gologger/levels"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/internal/runner/options.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolinit"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/headless/engine"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless/engine"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonexporter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonl"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/markdown"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/sarif"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/extensions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/yaml"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:internal/runner/options.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
"github.com/projectdiscovery/utils/generic"
logutil "github.com/projectdiscovery/utils/log"
@@ -68,6 +91,7 @@ func ParseOptions(options *types.Options) {
if options.ShowVarDump {
vardump.EnableVarDump = true
+ vardump.Limit = options.VarDumpLimit
}
if options.ShowActions {
gologger.Info().Msgf("Showing available headless actions: ")
@@ -303,10 +327,17 @@ func createReportingOptions(options *types.Options) (*reporting.Options, error)
OmitRaw: options.OmitRawRequests,
}
}
+ // Combine options.
if options.JSONLExport != "" {
- reportingOptions.JSONLExporter = &jsonl.Options{
- File: options.JSONLExport,
- OmitRaw: options.OmitRawRequests,
+ // Combine the CLI options with the config file options with the CLI options taking precedence
+ if reportingOptions.JSONLExporter != nil {
+ reportingOptions.JSONLExporter.File = options.JSONLExport
+ reportingOptions.JSONLExporter.OmitRaw = options.OmitRawRequests
+ } else {
+ reportingOptions.JSONLExporter = &jsonl.Options{
+ File: options.JSONLExport,
+ OmitRaw: options.OmitRawRequests,
+ }
}
}
@@ -316,9 +347,6 @@ func createReportingOptions(options *types.Options) (*reporting.Options, error)
// configureOutput configures the output logging levels to be displayed on the screen
func configureOutput(options *types.Options) {
- // disable standard logger (ref: https://github.com/golang/go/issues/19895)
- defer logutil.DisableDefaultLogger()
-
if options.NoColor {
gologger.DefaultLogger.SetFormatter(formatter.NewCLI(true))
}
@@ -337,6 +365,9 @@ func configureOutput(options *types.Options) {
if options.Silent {
gologger.DefaultLogger.SetMaxLevel(levels.LevelSilent)
}
+
+ // disable standard logger (ref: https://github.com/golang/go/issues/19895)
+ logutil.DisableDefaultLogger()
}
// loadResolvers loads resolvers from both user-provided flags and file
diff --git a/internal/runner/proxy.go b/internal/runner/proxy.go
index 4905522ba7..abf62f49e3 100644
--- a/internal/runner/proxy.go
+++ b/internal/runner/proxy.go
@@ -9,6 +9,14 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/internal/runner/proxy.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:internal/runner/proxy.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
proxyutils "github.com/projectdiscovery/utils/proxy"
diff --git a/internal/runner/runner.go b/internal/runner/runner.go
index 98f864ff05..9fa0a4d4d5 100644
--- a/internal/runner/runner.go
+++ b/internal/runner/runner.go
@@ -13,12 +13,22 @@ import (
"sync/atomic"
"time"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/internal/pdcp"
"github.com/Explorer1092/nuclei/v3/pkg/authprovider"
"github.com/Explorer1092/nuclei/v3/pkg/input/provider"
"github.com/Explorer1092/nuclei/v3/pkg/installer"
"github.com/Explorer1092/nuclei/v3/pkg/loader/parser"
"github.com/Explorer1092/nuclei/v3/pkg/scan/events"
+=======
+ "github.com/projectdiscovery/nuclei/v3/internal/pdcp"
+ "github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz/frequency"
+ "github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
+ "github.com/projectdiscovery/nuclei/v3/pkg/installer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/loader/parser"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan/events"
+>>>>>>> projectdiscovery-main
uncoverlib "github.com/projectdiscovery/uncover"
pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
"github.com/projectdiscovery/utils/env"
@@ -61,6 +71,40 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/utils/stats"
"github.com/Explorer1092/nuclei/v3/pkg/utils/yaml"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/internal/colorizer"
+ "github.com/projectdiscovery/nuclei/v3/internal/httpapi"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
+ "github.com/projectdiscovery/nuclei/v3/pkg/core"
+ "github.com/projectdiscovery/nuclei/v3/pkg/external/customtemplates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/input"
+ parsers "github.com/projectdiscovery/nuclei/v3/pkg/loader/workflow"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/progress"
+ "github.com/projectdiscovery/nuclei/v3/pkg/projectfile"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/automaticscan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/globalmatchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hosterrorscache"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/uncover"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/excludematchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless/engine"
+ httpProtocol "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/httpclientpool"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/stats"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/yaml"
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
ptrutil "github.com/projectdiscovery/utils/ptr"
)
@@ -74,21 +118,22 @@ var (
// Runner is a client for running the enumeration process.
type Runner struct {
- output output.Writer
- interactsh *interactsh.Client
- options *types.Options
- projectFile *projectfile.ProjectFile
- catalog catalog.Catalog
- progress progress.Progress
- colorizer aurora.Aurora
- issuesClient reporting.Client
- browser *engine.Browser
- rateLimiter *ratelimit.Limiter
- hostErrors hosterrorscache.CacheInterface
- resumeCfg *types.ResumeCfg
- pprofServer *http.Server
- pdcpUploadErrMsg string
- inputProvider provider.InputProvider
+ output output.Writer
+ interactsh *interactsh.Client
+ options *types.Options
+ projectFile *projectfile.ProjectFile
+ catalog catalog.Catalog
+ progress progress.Progress
+ colorizer aurora.Aurora
+ issuesClient reporting.Client
+ browser *engine.Browser
+ rateLimiter *ratelimit.Limiter
+ hostErrors hosterrorscache.CacheInterface
+ resumeCfg *types.ResumeCfg
+ pprofServer *http.Server
+ pdcpUploadErrMsg string
+ inputProvider provider.InputProvider
+ fuzzFrequencyCache *frequency.Tracker
//general purpose temporary directory
tmpDir string
parser parser.Parser
@@ -359,6 +404,10 @@ func (r *Runner) runStandardEnumeration(executerOpts protocols.ExecutorOptions,
// Close releases all the resources and cleans up
func (r *Runner) Close() {
+ // dump hosterrors cache
+ if r.hostErrors != nil {
+ r.hostErrors.Close()
+ }
if r.output != nil {
r.output.Close()
}
@@ -385,6 +434,9 @@ func (r *Runner) Close() {
if r.tmpDir != "" {
_ = os.RemoveAll(r.tmpDir)
}
+
+ //this is no-op unless nuclei is built with stats build tag
+ events.Close()
}
// setupPDCPUpload sets up the PDCP upload writer
@@ -420,6 +472,9 @@ func (r *Runner) setupPDCPUpload(writer output.Writer) output.Writer {
if r.options.ScanName != "" {
uploadWriter.SetScanName(r.options.ScanName)
}
+ if r.options.TeamID != "" {
+ uploadWriter.SetTeamID(r.options.TeamID)
+ }
return output.NewMultiWriter(writer, uploadWriter)
}
@@ -444,27 +499,32 @@ func (r *Runner) RunEnumeration() error {
r.options.ExcludedTemplates = append(r.options.ExcludedTemplates, ignoreFile.Files...)
}
+ fuzzFreqCache := frequency.New(frequency.DefaultMaxTrackCount, r.options.FuzzParamFrequency)
+ r.fuzzFrequencyCache = fuzzFreqCache
+
// Create the executor options which will be used throughout the execution
// stage by the nuclei engine modules.
executorOpts := protocols.ExecutorOptions{
- Output: r.output,
- Options: r.options,
- Progress: r.progress,
- Catalog: r.catalog,
- IssuesClient: r.issuesClient,
- RateLimiter: r.rateLimiter,
- Interactsh: r.interactsh,
- ProjectFile: r.projectFile,
- Browser: r.browser,
- Colorizer: r.colorizer,
- ResumeCfg: r.resumeCfg,
- ExcludeMatchers: excludematchers.New(r.options.ExcludeMatchers),
- InputHelper: input.NewHelper(),
- TemporaryDirectory: r.tmpDir,
- Parser: r.parser,
- }
-
- if env.GetEnvOrDefault("NUCLEI_ARGS", "") == "req_url_pattern=true" {
+ Output: r.output,
+ Options: r.options,
+ Progress: r.progress,
+ Catalog: r.catalog,
+ IssuesClient: r.issuesClient,
+ RateLimiter: r.rateLimiter,
+ Interactsh: r.interactsh,
+ ProjectFile: r.projectFile,
+ Browser: r.browser,
+ Colorizer: r.colorizer,
+ ResumeCfg: r.resumeCfg,
+ ExcludeMatchers: excludematchers.New(r.options.ExcludeMatchers),
+ InputHelper: input.NewHelper(),
+ TemporaryDirectory: r.tmpDir,
+ Parser: r.parser,
+ FuzzParamsFrequency: fuzzFreqCache,
+ GlobalMatchers: globalmatchers.New(),
+ }
+
+ if config.DefaultConfig.IsDebugArgEnabled(config.DebugExportURLPattern) {
// Go StdLib style experimental/debug feature switch
executorOpts.ExportReqURLPattern = true
}
@@ -488,8 +548,17 @@ func (r *Runner) RunEnumeration() error {
}
if r.options.ShouldUseHostError() {
- cache := hosterrorscache.New(r.options.MaxHostError, hosterrorscache.DefaultMaxHostsCount, r.options.TrackError)
+ maxHostError := r.options.MaxHostError
+ if r.options.TemplateThreads > maxHostError {
+ gologger.Print().Msgf("[%v] The concurrency value is higher than max-host-error", r.colorizer.BrightYellow("WRN"))
+ gologger.Info().Msgf("Adjusting max-host-error to the concurrency value: %d", r.options.TemplateThreads)
+
+ maxHostError = r.options.TemplateThreads
+ }
+
+ cache := hosterrorscache.New(maxHostError, hosterrorscache.DefaultMaxHostsCount, r.options.TrackError)
cache.SetVerbose(r.options.Verbose)
+
r.hostErrors = cache
executorOpts.HostErrorsCache = cache
}
@@ -615,6 +684,7 @@ func (r *Runner) RunEnumeration() error {
if executorOpts.InputHelper != nil {
_ = executorOpts.InputHelper.Close()
}
+ r.fuzzFrequencyCache.Close()
// todo: error propagation without canonical straight error check is required by cloud?
// use safe dereferencing to avoid potential panics in case of previous unchecked errors
@@ -702,6 +772,8 @@ func (r *Runner) displayExecutionInfo(store *loader.Store) {
stats.ForceDisplayWarning(templates.ExcludedCodeTmplStats)
stats.ForceDisplayWarning(templates.ExludedDastTmplStats)
stats.ForceDisplayWarning(templates.TemplatesExcludedStats)
+ stats.ForceDisplayWarning(templates.ExcludedFileStats)
+ stats.ForceDisplayWarning(templates.ExcludedSelfContainedStats)
}
if tmplCount == 0 && workflowCount == 0 {
@@ -770,6 +842,52 @@ func (r *Runner) SaveResumeConfig(path string) error {
return os.WriteFile(path, data, permissionutil.ConfigFilePermission)
}
+// upload existing scan results to cloud with progress
+func UploadResultsToCloud(options *types.Options) error {
+ h := &pdcpauth.PDCPCredHandler{}
+ creds, err := h.GetCreds()
+ if err != nil {
+ return errors.Wrap(err, "could not get credentials for cloud upload")
+ }
+ ctx := context.TODO()
+ uploadWriter, err := pdcp.NewUploadWriter(ctx, creds)
+ if err != nil {
+ return errors.Wrap(err, "could not create upload writer")
+ }
+ if options.ScanID != "" {
+ _ = uploadWriter.SetScanID(options.ScanID)
+ }
+ if options.ScanName != "" {
+ uploadWriter.SetScanName(options.ScanName)
+ }
+ if options.TeamID != "" {
+ uploadWriter.SetTeamID(options.TeamID)
+ }
+
+ // Open file to count the number of results first
+ file, err := os.Open(options.ScanUploadFile)
+ if err != nil {
+ return errors.Wrap(err, "could not open scan upload file")
+ }
+ defer file.Close()
+
+ gologger.Info().Msgf("Uploading scan results to cloud dashboard from %s", options.ScanUploadFile)
+ dec := json.NewDecoder(file)
+ for dec.More() {
+ var r output.ResultEvent
+ err := dec.Decode(&r)
+ if err != nil {
+ gologger.Warning().Msgf("Could not decode jsonl: %s\n", err)
+ continue
+ }
+ if err = uploadWriter.Write(&r); err != nil {
+ gologger.Warning().Msgf("[%s] failed to upload: %s\n", r.TemplateID, err)
+ }
+ }
+ uploadWriter.Close()
+ return nil
+}
+
type WalkFunc func(reflect.Value, reflect.StructField)
// Walk traverses a struct and executes a callback function on each value in the struct.
diff --git a/internal/runner/runner_test.go b/internal/runner/runner_test.go
index 511597abea..0b5e11fb20 100644
--- a/internal/runner/runner_test.go
+++ b/internal/runner/runner_test.go
@@ -6,8 +6,18 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/internal/runner/runner_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:internal/runner/runner_test.go
+>>>>>>> projectdiscovery-main
)
func TestCreateReportingOptions(t *testing.T) {
diff --git a/internal/runner/templates.go b/internal/runner/templates.go
index 389589184b..4e25ed0eff 100644
--- a/internal/runner/templates.go
+++ b/internal/runner/templates.go
@@ -11,10 +11,30 @@ import (
"github.com/alecthomas/chroma/quick"
jsoniter "github.com/json-iterator/go"
"github.com/logrusorgru/aurora"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/internal/runner/templates.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/loader"
+>>>>>>> projectdiscovery-main
"github.com/Explorer1092/nuclei/v3/pkg/templates"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/Explorer1092/nuclei/v2/pkg/parsers"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
+
+ "github.com/projectdiscovery/gologger"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:internal/runner/templates.go
+>>>>>>> projectdiscovery-main
)
// log available templates for verbose (-vv)
diff --git a/lib/config.go b/lib/config.go
index d011e17dff..27c44e540e 100644
--- a/lib/config.go
+++ b/lib/config.go
@@ -164,11 +164,17 @@ func WithConcurrency(opts Concurrency) NucleiSDKOptions {
}
// WithGlobalRateLimit sets global rate (i.e all hosts combined) limit options
+// Deprecated: will be removed in favour of WithGlobalRateLimitCtx in next release
func WithGlobalRateLimit(maxTokens int, duration time.Duration) NucleiSDKOptions {
+ return WithGlobalRateLimitCtx(context.Background(), maxTokens, duration)
+}
+
+// WithGlobalRateLimitCtx allows setting a global rate limit for the entire engine
+func WithGlobalRateLimitCtx(ctx context.Context, maxTokens int, duration time.Duration) NucleiSDKOptions {
return func(e *NucleiEngine) error {
e.opts.RateLimit = maxTokens
e.opts.RateLimitDuration = duration
- e.rateLimiter = ratelimit.New(context.Background(), uint(e.opts.RateLimit), e.opts.RateLimitDuration)
+ e.rateLimiter = ratelimit.New(ctx, uint(e.opts.RateLimit), e.opts.RateLimitDuration)
return nil
}
}
@@ -374,6 +380,23 @@ func WithSandboxOptions(allowLocalFileAccess bool, restrictLocalNetworkAccess bo
func EnableCodeTemplates() NucleiSDKOptions {
return func(e *NucleiEngine) error {
e.opts.EnableCodeTemplates = true
+ e.opts.EnableSelfContainedTemplates = true
+ return nil
+ }
+}
+
+// EnableSelfContainedTemplates allows loading/executing self-contained templates
+func EnableSelfContainedTemplates() NucleiSDKOptions {
+ return func(e *NucleiEngine) error {
+ e.opts.EnableSelfContainedTemplates = true
+ return nil
+ }
+}
+
+// EnableFileTemplates allows loading/executing file protocol templates
+func EnableFileTemplates() NucleiSDKOptions {
+ return func(e *NucleiEngine) error {
+ e.opts.EnableFileTemplates = true
return nil
}
}
@@ -386,10 +409,30 @@ func WithHeaders(headers []string) NucleiSDKOptions {
}
}
+// WithVars allows setting custom variables to use in templates/workflows context
+func WithVars(vars []string) NucleiSDKOptions {
+ // Create a goflags.RuntimeMap
+ runtimeVars := goflags.RuntimeMap{}
+ for _, v := range vars {
+ err := runtimeVars.Set(v)
+ if err != nil {
+ return func(e *NucleiEngine) error {
+ return err
+ }
+ }
+ }
+
+ return func(e *NucleiEngine) error {
+ e.opts.Vars = runtimeVars
+ return nil
+ }
+}
+
// EnablePassiveMode allows enabling passive HTTP response processing mode
func EnablePassiveMode() NucleiSDKOptions {
return func(e *NucleiEngine) error {
e.opts.OfflineHTTP = true
+ e.opts.DisableHTTPProbe = true
return nil
}
}
@@ -434,3 +477,11 @@ func WithCatalog(cat catalog.Catalog) NucleiSDKOptions {
return nil
}
}
+
+// DisableUpdateCheck disables nuclei update check
+func DisableUpdateCheck() NucleiSDKOptions {
+ return func(e *NucleiEngine) error {
+ DefaultConfig.DisableUpdateCheck()
+ return nil
+ }
+}
diff --git a/lib/example_test.go b/lib/example_test.go
index 7c14702a77..5154577f14 100644
--- a/lib/example_test.go
+++ b/lib/example_test.go
@@ -7,7 +7,12 @@ import (
"os"
"testing"
+<<<<<<< HEAD
nuclei "github.com/Explorer1092/nuclei/v3/lib"
+=======
+ "github.com/kitabisa/go-ci"
+ nuclei "github.com/projectdiscovery/nuclei/v3/lib"
+>>>>>>> projectdiscovery-main
"github.com/remeh/sizedwaitgroup"
)
@@ -78,9 +83,10 @@ func ExampleThreadSafeNucleiEngine() {
func TestMain(m *testing.M) {
// this file only contains testtables examples https://go.dev/blog/examples
// and actual functionality test are in sdk_test.go
- if os.Getenv("GH_ACTION") != "" || os.Getenv("CI") != "" {
+ if ci.IsCI() {
// no need to run this test on github actions
return
}
- m.Run()
+
+ os.Exit(m.Run())
}
diff --git a/lib/multi.go b/lib/multi.go
index ee78bf4912..60c473e43d 100644
--- a/lib/multi.go
+++ b/lib/multi.go
@@ -26,7 +26,7 @@ type unsafeOptions struct {
}
// createEphemeralObjects creates ephemeral nuclei objects/instances/types
-func createEphemeralObjects(base *NucleiEngine, opts *types.Options) (*unsafeOptions, error) {
+func createEphemeralObjects(ctx context.Context, base *NucleiEngine, opts *types.Options) (*unsafeOptions, error) {
u := &unsafeOptions{}
u.executerOpts = protocols.ExecutorOptions{
Output: base.customWriter,
@@ -49,9 +49,9 @@ func createEphemeralObjects(base *NucleiEngine, opts *types.Options) (*unsafeOpt
opts.RateLimitDuration = time.Second
}
if opts.RateLimit == 0 && opts.RateLimitDuration == 0 {
- u.executerOpts.RateLimiter = ratelimit.NewUnlimited(context.Background())
+ u.executerOpts.RateLimiter = ratelimit.NewUnlimited(ctx)
} else {
- u.executerOpts.RateLimiter = ratelimit.New(context.Background(), uint(opts.RateLimit), opts.RateLimitDuration)
+ u.executerOpts.RateLimiter = ratelimit.New(ctx, uint(opts.RateLimit), opts.RateLimitDuration)
}
u.engine = core.New(opts)
u.engine.SetExecuterOptions(u.executerOpts)
@@ -83,7 +83,7 @@ type ThreadSafeNucleiEngine struct {
// NewThreadSafeNucleiEngine creates a new nuclei engine with given options
// whose methods are thread-safe and can be used concurrently
// Note: Non-thread-safe methods start with Global prefix
-func NewThreadSafeNucleiEngine(opts ...NucleiSDKOptions) (*ThreadSafeNucleiEngine, error) {
+func NewThreadSafeNucleiEngineCtx(ctx context.Context, opts ...NucleiSDKOptions) (*ThreadSafeNucleiEngine, error) {
// default options
e := &NucleiEngine{
opts: types.DefaultOptions(),
@@ -94,12 +94,17 @@ func NewThreadSafeNucleiEngine(opts ...NucleiSDKOptions) (*ThreadSafeNucleiEngin
return nil, err
}
}
- if err := e.init(); err != nil {
+ if err := e.init(ctx); err != nil {
return nil, err
}
return &ThreadSafeNucleiEngine{eng: e}, nil
}
+// Deprecated: use NewThreadSafeNucleiEngineCtx instead
+func NewThreadSafeNucleiEngine(opts ...NucleiSDKOptions) (*ThreadSafeNucleiEngine, error) {
+ return NewThreadSafeNucleiEngineCtx(context.Background(), opts...)
+}
+
// GlobalLoadAllTemplates loads all templates from nuclei-templates repo
// This method will load all templates based on filters given at the time of nuclei engine creation in opts
func (e *ThreadSafeNucleiEngine) GlobalLoadAllTemplates() error {
@@ -123,9 +128,9 @@ func (e *ThreadSafeNucleiEngine) ExecuteNucleiWithOptsCtx(ctx context.Context, t
return err
}
}
- defer tmpEngine.Close()
+
// create ephemeral nuclei objects/instances/types using base nuclei engine
- unsafeOpts, err := createEphemeralObjects(e.eng, tmpEngine.opts)
+ unsafeOpts, err := createEphemeralObjects(ctx, e.eng, tmpEngine.opts)
if err != nil {
return err
}
@@ -157,7 +162,7 @@ func (e *ThreadSafeNucleiEngine) ExecuteNucleiWithOptsCtx(ctx context.Context, t
engine := core.New(tmpEngine.opts)
engine.SetExecuterOptions(unsafeOpts.executerOpts)
- _ = engine.ExecuteScanWithOpts(context.Background(), store.Templates(), inputProvider, false)
+ _ = engine.ExecuteScanWithOpts(ctx, store.Templates(), inputProvider, false)
engine.WorkPool().Wait()
return nil
diff --git a/lib/sdk.go b/lib/sdk.go
index c07fc119c5..97765a6b0d 100644
--- a/lib/sdk.go
+++ b/lib/sdk.go
@@ -6,6 +6,7 @@ import (
"context"
"io"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/authprovider"
"github.com/Explorer1092/nuclei/v3/pkg/catalog"
"github.com/Explorer1092/nuclei/v3/pkg/catalog/loader"
@@ -25,6 +26,26 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/templates"
"github.com/Explorer1092/nuclei/v3/pkg/templates/signer"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
+ "github.com/projectdiscovery/nuclei/v3/pkg/core"
+ "github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
+ providerTypes "github.com/projectdiscovery/nuclei/v3/pkg/input/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/loader/workflow"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/progress"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hosterrorscache"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless/engine"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/signer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/ratelimit"
"github.com/projectdiscovery/retryablehttp-go"
errorutil "github.com/projectdiscovery/utils/errors"
@@ -101,6 +122,7 @@ func (e *NucleiEngine) LoadAllTemplates() error {
return errorutil.New("Could not create loader client: %s\n", err)
}
e.store.Load()
+ e.templatesLoaded = true
return nil
}
@@ -112,7 +134,19 @@ func (e *NucleiEngine) GetTemplates() []*templates.Template {
return e.store.Templates()
}
+<<<<<<< HEAD
// LoadTargets LoadTargets(urls/domains/ips only) adds targets to the nuclei engine
+=======
+// GetWorkflows returns all nuclei workflows that are loaded
+func (e *NucleiEngine) GetWorkflows() []*templates.Template {
+ if !e.templatesLoaded {
+ _ = e.LoadAllTemplates()
+ }
+ return e.store.Workflows()
+}
+
+// LoadTargets(urls/domains/ips only) adds targets to the nuclei engine
+>>>>>>> projectdiscovery-main
func (e *NucleiEngine) LoadTargets(targets []string, probeNonHttp bool) {
for _, target := range targets {
if probeNonHttp {
@@ -203,13 +237,13 @@ func (e *NucleiEngine) SignTemplate(tmplSigner *signer.TemplateSigner, data []by
if err != nil {
return data, err
}
- buff := bytes.NewBuffer(signer.RemoveSignatureFromData(data))
+ _, content := signer.ExtractSignatureAndContent(data)
+ buff := bytes.NewBuffer(content)
buff.WriteString("\n" + signatureData)
return buff.Bytes(), err
}
-// Close all resources used by nuclei engine
-func (e *NucleiEngine) Close() {
+func (e *NucleiEngine) closeInternal() {
if e.interactshClient != nil {
e.interactshClient.Close()
}
@@ -231,8 +265,6 @@ func (e *NucleiEngine) Close() {
if e.rateLimiter != nil {
e.rateLimiter.Stop()
}
- // close global shared resources
- protocolstate.Close()
if e.inputProvider != nil {
e.inputProvider.Close()
}
@@ -244,6 +276,12 @@ func (e *NucleiEngine) Close() {
}
}
+// Close all resources used by nuclei engine
+func (e *NucleiEngine) Close() {
+ e.closeInternal()
+ protocolinit.Close()
+}
+
// ExecuteCallbackWithCtx executes templates on targets and calls callback on each result(only if results are found)
// enable matcher-status option if you expect this callback to be called for all results regardless if it matched or not
func (e *NucleiEngine) ExecuteCallbackWithCtx(ctx context.Context, callback ...func(event *output.ResultEvent)) error {
@@ -265,7 +303,12 @@ func (e *NucleiEngine) ExecuteCallbackWithCtx(ctx context.Context, callback ...f
}
e.resultCallbacks = append(e.resultCallbacks, filtered...)
- _ = e.engine.ExecuteScanWithOpts(context.Background(), e.store.Templates(), e.inputProvider, false)
+ templatesAndWorkflows := append(e.store.Templates(), e.store.Workflows()...)
+ if len(templatesAndWorkflows) == 0 {
+ return ErrNoTemplatesAvailable
+ }
+
+ _ = e.engine.ExecuteScanWithOpts(ctx, templatesAndWorkflows, e.inputProvider, false)
defer e.engine.WorkPool().Wait()
return nil
}
@@ -286,8 +329,13 @@ func (e *NucleiEngine) Engine() *core.Engine {
return e.engine
}
-// NewNucleiEngine creates a new nuclei engine instance
-func NewNucleiEngine(options ...NucleiSDKOptions) (*NucleiEngine, error) {
+// Store returns store of nuclei
+func (e *NucleiEngine) Store() *loader.Store {
+ return e.store
+}
+
+// NewNucleiEngineCtx creates a new nuclei engine instance with given context
+func NewNucleiEngineCtx(ctx context.Context, options ...NucleiSDKOptions) (*NucleiEngine, error) {
// default options
e := &NucleiEngine{
opts: types.DefaultOptions(),
@@ -298,8 +346,13 @@ func NewNucleiEngine(options ...NucleiSDKOptions) (*NucleiEngine, error) {
return nil, err
}
}
- if err := e.init(); err != nil {
+ if err := e.init(ctx); err != nil {
return nil, err
}
return e, nil
}
+
+// Deprecated: use NewNucleiEngineCtx instead
+func NewNucleiEngine(options ...NucleiSDKOptions) (*NucleiEngine, error) {
+ return NewNucleiEngineCtx(context.Background(), options...)
+}
diff --git a/lib/sdk_private.go b/lib/sdk_private.go
index 410c059feb..38bbd01f2d 100644
--- a/lib/sdk_private.go
+++ b/lib/sdk_private.go
@@ -7,6 +7,7 @@ import (
"sync"
"time"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/internal/runner"
"github.com/Explorer1092/nuclei/v3/pkg/authprovider"
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
@@ -27,6 +28,10 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
"github.com/Explorer1092/nuclei/v3/pkg/types"
nucleiUtils "github.com/Explorer1092/nuclei/v3/pkg/utils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/input"
+
+>>>>>>> projectdiscovery-main
"github.com/logrusorgru/aurora"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
@@ -35,10 +40,10 @@ import (
"github.com/projectdiscovery/ratelimit"
)
-var sharedInit sync.Once = sync.Once{}
+var sharedInit *sync.Once
// applyRequiredDefaults to options
-func (e *NucleiEngine) applyRequiredDefaults() {
+func (e *NucleiEngine) applyRequiredDefaults(ctx context.Context) {
mockoutput := testutils.NewMockOutputWriter(e.opts.OmitTemplate)
mockoutput.WriteCallback = func(event *output.ResultEvent) {
if len(e.resultCallbacks) > 0 {
@@ -82,7 +87,7 @@ func (e *NucleiEngine) applyRequiredDefaults() {
e.interactshOpts = interactsh.DefaultOptions(e.customWriter, e.rc, e.customProgress)
}
if e.rateLimiter == nil {
- e.rateLimiter = ratelimit.New(context.Background(), 150, time.Second)
+ e.rateLimiter = ratelimit.New(ctx, 150, time.Second)
}
if e.opts.ExcludeTags == nil {
e.opts.ExcludeTags = []string{}
@@ -95,7 +100,7 @@ func (e *NucleiEngine) applyRequiredDefaults() {
}
// init
-func (e *NucleiEngine) init() error {
+func (e *NucleiEngine) init(ctx context.Context) error {
if e.opts.Verbose {
gologger.DefaultLogger.SetMaxLevel(levels.LevelVerbose)
} else if e.opts.Debug {
@@ -108,6 +113,16 @@ func (e *NucleiEngine) init() error {
return err
}
+ e.parser = templates.NewParser()
+
+ if sharedInit == nil || protocolstate.ShouldInit() {
+ sharedInit = &sync.Once{}
+ }
+
+ sharedInit.Do(func() {
+ _ = protocolinit.Init(e.opts)
+ })
+
if e.opts.ProxyInternal && types.ProxyURL != "" || types.ProxySocksURL != "" {
httpclient, err := httpclientpool.Get(e.opts, &httpclientpool.Configuration{})
if err != nil {
@@ -116,14 +131,7 @@ func (e *NucleiEngine) init() error {
e.httpClient = httpclient
}
- e.parser = templates.NewParser()
-
- sharedInit.Do(func() {
- _ = protocolstate.Init(e.opts)
- _ = protocolinit.Init(e.opts)
- })
-
- e.applyRequiredDefaults()
+ e.applyRequiredDefaults(ctx)
var err error
// setup progressbar
@@ -168,6 +176,7 @@ func (e *NucleiEngine) init() error {
ResumeCfg: types.NewResumeCfg(),
Browser: e.browserInstance,
Parser: e.parser,
+ InputHelper: input.NewHelper(),
}
if len(e.opts.SecretsFile) > 0 {
authTmplStore, err := runner.GetAuthTmplStore(*e.opts, e.catalog, e.executerOpts)
@@ -206,9 +215,9 @@ func (e *NucleiEngine) init() error {
e.opts.RateLimitDuration = time.Second
}
if e.opts.RateLimit == 0 && e.opts.RateLimitDuration == 0 {
- e.executerOpts.RateLimiter = ratelimit.NewUnlimited(context.Background())
+ e.executerOpts.RateLimiter = ratelimit.NewUnlimited(ctx)
} else {
- e.executerOpts.RateLimiter = ratelimit.New(context.Background(), uint(e.opts.RateLimit), e.opts.RateLimitDuration)
+ e.executerOpts.RateLimiter = ratelimit.New(ctx, uint(e.opts.RateLimit), e.opts.RateLimitDuration)
}
}
@@ -228,7 +237,10 @@ func (e *NucleiEngine) init() error {
// and also upgrade templates to latest version if available
installer.NucleiSDKVersionCheck()
- return e.processUpdateCheckResults()
+ if DefaultConfig.CanCheckForUpdates() {
+ return e.processUpdateCheckResults()
+ }
+ return nil
}
type syncOnce struct {
diff --git a/lib/tests/sdk_test.go b/lib/tests/sdk_test.go
index 37771120c7..000065cb4f 100644
--- a/lib/tests/sdk_test.go
+++ b/lib/tests/sdk_test.go
@@ -1,6 +1,7 @@
package sdk_test
import (
+ "context"
"os"
"os/exec"
"testing"
@@ -28,8 +29,9 @@ func TestSimpleNuclei(t *testing.T) {
time.Sleep(2 * time.Second)
goleak.VerifyNone(t, knownLeaks...)
}()
- ne, err := nuclei.NewNucleiEngine(
- nuclei.WithTemplateFilters(nuclei.TemplateFilters{ProtocolTypes: "dns"}),
+ ne, err := nuclei.NewNucleiEngineCtx(
+ context.TODO(),
+ nuclei.WithTemplateFilters(nuclei.TemplateFilters{ProtocolTypes: "dns"}), // filter dns templates
nuclei.EnableStatsWithOpts(nuclei.StatsOptions{JSON: true}),
)
require.Nil(t, err)
@@ -62,7 +64,8 @@ func TestSimpleNucleiRemote(t *testing.T) {
time.Sleep(2 * time.Second)
goleak.VerifyNone(t, knownLeaks...)
}()
- ne, err := nuclei.NewNucleiEngine(
+ ne, err := nuclei.NewNucleiEngineCtx(
+ context.TODO(),
nuclei.WithTemplatesOrWorkflows(
nuclei.TemplateSources{
RemoteTemplates: []string{"https://cloud.projectdiscovery.io/public/nameserver-fingerprint.yaml"},
@@ -100,7 +103,7 @@ func TestThreadSafeNuclei(t *testing.T) {
goleak.VerifyNone(t, knownLeaks...)
}()
// create nuclei engine with options
- ne, err := nuclei.NewThreadSafeNucleiEngine()
+ ne, err := nuclei.NewThreadSafeNucleiEngineCtx(context.TODO())
require.Nil(t, err)
// scan 1 = run dns templates on scanme.sh
@@ -130,3 +133,37 @@ func TestThreadSafeNuclei(t *testing.T) {
fn()
}
}
+
+func TestWithVarsNuclei(t *testing.T) {
+ fn := func() {
+ defer func() {
+ // resources like leveldb have a delay to commit in-memory resources
+ // to disk, typically 1-2 seconds, so we wait for 2 seconds
+ time.Sleep(2 * time.Second)
+ goleak.VerifyNone(t, knownLeaks...)
+ }()
+ ne, err := nuclei.NewNucleiEngineCtx(
+ context.TODO(),
+ nuclei.EnableSelfContainedTemplates(),
+ nuclei.WithTemplatesOrWorkflows(nuclei.TemplateSources{Templates: []string{"http/token-spray/api-1forge.yaml"}}),
+ nuclei.WithVars([]string{"token=foobar"}),
+ nuclei.WithVerbosity(nuclei.VerbosityOptions{Debug: true}),
+ )
+ require.Nil(t, err)
+ ne.LoadTargets([]string{"scanme.sh"}, true) // probe http/https target is set to true here
+ err = ne.ExecuteWithCallback(nil)
+ require.Nil(t, err)
+ defer ne.Close()
+ }
+ // this is shared test so needs to be run as seperate process
+ if env.GetEnvOrDefault("TestWithVarsNuclei", false) {
+ cmd := exec.Command(os.Args[0], "-test.run=TestWithVarsNuclei")
+ cmd.Env = append(os.Environ(), "TestWithVarsNuclei=true")
+ out, err := cmd.CombinedOutput()
+ if err != nil {
+ t.Fatalf("process ran with error %s, output: %s", err, out)
+ }
+ } else {
+ fn()
+ }
+}
diff --git a/nuclei-jsonschema.json b/nuclei-jsonschema.json
index 46e1ab986d..ca1164ccb6 100644
--- a/nuclei-jsonschema.json
+++ b/nuclei-jsonschema.json
@@ -3,6 +3,22 @@
"$id": "https://templates.-template",
"$ref": "#/$defs/templates.Template",
"$defs": {
+ "analyzers.AnalyzerTemplate": {
+ "properties": {
+ "name": {
+ "type": "string"
+ },
+ "parameters": {
+ "$ref": "#/$defs/map[string]interface {}"
+ }
+ },
+ "additionalProperties": false,
+ "type": "object",
+ "required": [
+ "name",
+ "parameters"
+ ]
+ },
"code.Request": {
"properties": {
"matchers": {
@@ -247,7 +263,12 @@
"time",
"select",
"files",
+ "waitdom",
+ "waitfcp",
+ "waitfmp",
+ "waitidle",
"waitload",
+ "waitstable",
"getresource",
"extract",
"setmethod",
@@ -256,6 +277,7 @@
"deleteheader",
"setbody",
"waitevent",
+ "waitdialog",
"keyboard",
"debug",
"sleep",
@@ -457,6 +479,22 @@
"title": "part of rule",
"description": "Part of request rule to fuzz"
},
+ "parts": {
+ "items": {
+ "type": "string",
+ "enum": [
+ "query",
+ "header",
+ "path",
+ "body",
+ "cookie",
+ "request"
+ ]
+ },
+ "type": "array",
+ "title": "parts of rule",
+ "description": "Part of request rule to fuzz"
+ },
"mode": {
"type": "string",
"enum": [
@@ -763,6 +801,11 @@
"title": "fuzzin rules for http fuzzing",
"description": "Fuzzing describes rule schema to fuzz http requests"
},
+ "analyzer": {
+ "$ref": "#/$defs/analyzers.AnalyzerTemplate",
+ "title": "analyzer for http request",
+ "description": "Analyzer for HTTP Request"
+ },
"self-contained": {
"type": "boolean"
},
@@ -771,6 +814,11 @@
"title": "signature is the http request signature method",
"description": "Signature is the HTTP Request signature Method"
},
+ "skip-secret-file": {
+ "type": "boolean",
+ "title": "bypass secret file",
+ "description": "Skips the authentication or authorization configured in the secret file"
+ },
"cookie-reuse": {
"type": "boolean",
"title": "optional cookie reuse enable",
@@ -862,6 +910,11 @@
],
"title": "condition between the filters",
"description": "Operator to use between multiple per-conditions"
+ },
+ "global-matchers": {
+ "type": "boolean",
+ "title": "global matchers",
+ "description": "marks matchers as static and applies globally to all result events from other templates"
}
},
"additionalProperties": false,
@@ -925,11 +978,6 @@
"title": "code to execute in javascript",
"description": "Executes inline javascript code for the request"
},
- "timeout": {
- "type": "integer",
- "title": "timeout for javascript execution",
- "description": "Timeout in seconds is optional timeout for entire javascript script execution"
- },
"stop-at-first-match": {
"type": "boolean",
"title": "stop at first match",
@@ -1343,6 +1391,11 @@
"title": "read all response stream",
"description": "Read all response stream till the server stops sending"
},
+ "stop-at-first-match": {
+ "type": "boolean",
+ "title": "stop at first match",
+ "description": "Stop the execution after a match is found"
+ },
"matchers": {
"items": {
"$ref": "#/$defs/matchers.Matcher"
diff --git a/pkg/authprovider/file.go b/pkg/authprovider/file.go
index afdac7c226..56b6588fd5 100644
--- a/pkg/authprovider/file.go
+++ b/pkg/authprovider/file.go
@@ -16,8 +16,8 @@ import (
type FileAuthProvider struct {
Path string
store *authx.Authx
- compiled map[*regexp.Regexp]authx.AuthStrategy
- domains map[string]authx.AuthStrategy
+ compiled map[*regexp.Regexp][]authx.AuthStrategy
+ domains map[string][]authx.AuthStrategy
}
// NewFileAuthProvider creates a new file based auth provider
@@ -51,29 +51,36 @@ func NewFileAuthProvider(path string, callback authx.LazyFetchSecret) (AuthProvi
// init initializes the file auth provider
func (f *FileAuthProvider) init() {
- for _, secret := range f.store.Secrets {
+ for _, _secret := range f.store.Secrets {
+ secret := _secret // allocate copy of pointer
if len(secret.DomainsRegex) > 0 {
for _, domain := range secret.DomainsRegex {
if f.compiled == nil {
- f.compiled = make(map[*regexp.Regexp]authx.AuthStrategy)
+ f.compiled = make(map[*regexp.Regexp][]authx.AuthStrategy)
}
compiled, err := regexp.Compile(domain)
if err != nil {
continue
}
- f.compiled[compiled] = secret.GetStrategy()
+
+ if ss, ok := f.compiled[compiled]; ok {
+ f.compiled[compiled] = append(ss, secret.GetStrategy())
+ } else {
+ f.compiled[compiled] = []authx.AuthStrategy{secret.GetStrategy()}
+ }
}
}
for _, domain := range secret.Domains {
if f.domains == nil {
- f.domains = make(map[string]authx.AuthStrategy)
+ f.domains = make(map[string][]authx.AuthStrategy)
}
- f.domains[strings.TrimSpace(domain)] = secret.GetStrategy()
- if strings.HasSuffix(domain, ":80") {
- f.domains[strings.TrimSuffix(domain, ":80")] = secret.GetStrategy()
- }
- if strings.HasSuffix(domain, ":443") {
- f.domains[strings.TrimSuffix(domain, ":443")] = secret.GetStrategy()
+ domain = strings.TrimSpace(domain)
+ domain = strings.TrimSuffix(domain, ":80")
+ domain = strings.TrimSuffix(domain, ":443")
+ if ss, ok := f.domains[domain]; ok {
+ f.domains[domain] = append(ss, secret.GetStrategy())
+ } else {
+ f.domains[domain] = []authx.AuthStrategy{secret.GetStrategy()}
}
}
}
@@ -81,32 +88,38 @@ func (f *FileAuthProvider) init() {
if len(dynamic.DomainsRegex) > 0 {
for _, domain := range dynamic.DomainsRegex {
if f.compiled == nil {
- f.compiled = make(map[*regexp.Regexp]authx.AuthStrategy)
+ f.compiled = make(map[*regexp.Regexp][]authx.AuthStrategy)
}
compiled, err := regexp.Compile(domain)
if err != nil {
continue
}
- f.compiled[compiled] = &authx.DynamicAuthStrategy{Dynamic: dynamic}
+ if ss, ok := f.compiled[compiled]; !ok {
+ f.compiled[compiled] = []authx.AuthStrategy{&authx.DynamicAuthStrategy{Dynamic: dynamic}}
+ } else {
+ f.compiled[compiled] = append(ss, &authx.DynamicAuthStrategy{Dynamic: dynamic})
+ }
}
}
for _, domain := range dynamic.Domains {
if f.domains == nil {
- f.domains = make(map[string]authx.AuthStrategy)
+ f.domains = make(map[string][]authx.AuthStrategy)
}
- f.domains[strings.TrimSpace(domain)] = &authx.DynamicAuthStrategy{Dynamic: dynamic}
- if strings.HasSuffix(domain, ":80") {
- f.domains[strings.TrimSuffix(domain, ":80")] = &authx.DynamicAuthStrategy{Dynamic: dynamic}
- }
- if strings.HasSuffix(domain, ":443") {
- f.domains[strings.TrimSuffix(domain, ":443")] = &authx.DynamicAuthStrategy{Dynamic: dynamic}
+ domain = strings.TrimSpace(domain)
+ domain = strings.TrimSuffix(domain, ":80")
+ domain = strings.TrimSuffix(domain, ":443")
+
+ if ss, ok := f.domains[domain]; !ok {
+ f.domains[domain] = []authx.AuthStrategy{&authx.DynamicAuthStrategy{Dynamic: dynamic}}
+ } else {
+ f.domains[domain] = append(ss, &authx.DynamicAuthStrategy{Dynamic: dynamic})
}
}
}
}
// LookupAddr looks up a given domain/address and returns appropriate auth strategy
-func (f *FileAuthProvider) LookupAddr(addr string) authx.AuthStrategy {
+func (f *FileAuthProvider) LookupAddr(addr string) []authx.AuthStrategy {
if strings.Contains(addr, ":") {
// default normalization for host:port
host, port, err := net.SplitHostPort(addr)
@@ -128,12 +141,12 @@ func (f *FileAuthProvider) LookupAddr(addr string) authx.AuthStrategy {
}
// LookupURL looks up a given URL and returns appropriate auth strategy
-func (f *FileAuthProvider) LookupURL(u *url.URL) authx.AuthStrategy {
+func (f *FileAuthProvider) LookupURL(u *url.URL) []authx.AuthStrategy {
return f.LookupAddr(u.Host)
}
// LookupURLX looks up a given URL and returns appropriate auth strategy
-func (f *FileAuthProvider) LookupURLX(u *urlutil.URL) authx.AuthStrategy {
+func (f *FileAuthProvider) LookupURLX(u *urlutil.URL) []authx.AuthStrategy {
return f.LookupAddr(u.Host)
}
@@ -150,17 +163,21 @@ func (f *FileAuthProvider) GetTemplatePaths() []string {
// PreFetchSecrets pre-fetches the secrets from the auth provider
func (f *FileAuthProvider) PreFetchSecrets() error {
- for _, s := range f.domains {
- if val, ok := s.(*authx.DynamicAuthStrategy); ok {
- if err := val.Dynamic.Fetch(false); err != nil {
- return err
+ for _, ss := range f.domains {
+ for _, s := range ss {
+ if val, ok := s.(*authx.DynamicAuthStrategy); ok {
+ if err := val.Dynamic.Fetch(false); err != nil {
+ return err
+ }
}
}
}
- for _, s := range f.compiled {
- if val, ok := s.(*authx.DynamicAuthStrategy); ok {
- if err := val.Dynamic.Fetch(false); err != nil {
- return err
+ for _, ss := range f.compiled {
+ for _, s := range ss {
+ if val, ok := s.(*authx.DynamicAuthStrategy); ok {
+ if err := val.Dynamic.Fetch(false); err != nil {
+ return err
+ }
}
}
}
diff --git a/pkg/authprovider/interface.go b/pkg/authprovider/interface.go
index 6c6fe53c16..04eb8289fc 100644
--- a/pkg/authprovider/interface.go
+++ b/pkg/authprovider/interface.go
@@ -22,13 +22,13 @@ var (
type AuthProvider interface {
// LookupAddr looks up a given domain/address and returns appropriate auth strategy
// for it (accepted inputs are scanme.sh or scanme.sh:443)
- LookupAddr(string) authx.AuthStrategy
+ LookupAddr(string) []authx.AuthStrategy
// LookupURL looks up a given URL and returns appropriate auth strategy
// it accepts a valid url struct and returns the auth strategy
- LookupURL(*url.URL) authx.AuthStrategy
+ LookupURL(*url.URL) []authx.AuthStrategy
// LookupURLX looks up a given URL and returns appropriate auth strategy
// it accepts pd url struct (i.e urlutil.URL) and returns the auth strategy
- LookupURLX(*urlutil.URL) authx.AuthStrategy
+ LookupURLX(*urlutil.URL) []authx.AuthStrategy
// GetTemplatePaths returns the template path for the auth provider
// that will be used for dynamic secret fetching
GetTemplatePaths() []string
diff --git a/pkg/authprovider/multi.go b/pkg/authprovider/multi.go
index 244bebe16d..444bece904 100644
--- a/pkg/authprovider/multi.go
+++ b/pkg/authprovider/multi.go
@@ -19,7 +19,7 @@ func NewMultiAuthProvider(providers ...AuthProvider) AuthProvider {
return &MultiAuthProvider{Providers: providers}
}
-func (m *MultiAuthProvider) LookupAddr(host string) authx.AuthStrategy {
+func (m *MultiAuthProvider) LookupAddr(host string) []authx.AuthStrategy {
for _, provider := range m.Providers {
strategy := provider.LookupAddr(host)
if strategy != nil {
@@ -29,7 +29,7 @@ func (m *MultiAuthProvider) LookupAddr(host string) authx.AuthStrategy {
return nil
}
-func (m *MultiAuthProvider) LookupURL(u *url.URL) authx.AuthStrategy {
+func (m *MultiAuthProvider) LookupURL(u *url.URL) []authx.AuthStrategy {
for _, provider := range m.Providers {
strategy := provider.LookupURL(u)
if strategy != nil {
@@ -39,7 +39,7 @@ func (m *MultiAuthProvider) LookupURL(u *url.URL) authx.AuthStrategy {
return nil
}
-func (m *MultiAuthProvider) LookupURLX(u *urlutil.URL) authx.AuthStrategy {
+func (m *MultiAuthProvider) LookupURLX(u *urlutil.URL) []authx.AuthStrategy {
for _, provider := range m.Providers {
strategy := provider.LookupURLX(u)
if strategy != nil {
diff --git a/pkg/catalog/config/constants.go b/pkg/catalog/config/constants.go
index 736671127d..2fdd23dd45 100644
--- a/pkg/catalog/config/constants.go
+++ b/pkg/catalog/config/constants.go
@@ -31,7 +31,7 @@ const (
CLIConfigFileName = "config.yaml"
ReportingConfigFilename = "reporting-config.yaml"
// Version is the current version of nuclei
- Version = `v3.2.7`
+ Version = `v3.3.6`
// Directory Names of custom templates
CustomS3TemplatesDirName = "s3"
CustomGitHubTemplatesDirName = "github"
@@ -68,3 +68,13 @@ func trimDevIfExists(version string) string {
}
return version
}
+
+// similar to go pattern of enabling debug related features
+// we add custom/extra switches for debugging purposes
+const (
+ // DebugArgHostErrorStats is used to print host error stats
+ // when it is closed
+ DebugArgHostErrorStats = "host-error-stats"
+ // DebugExportReqURLPattern is used to export request URL pattern
+ DebugExportURLPattern = "req-url-pattern"
+)
diff --git a/pkg/catalog/config/nucleiconfig.go b/pkg/catalog/config/nucleiconfig.go
index 619d7fd47a..062393e542 100644
--- a/pkg/catalog/config/nucleiconfig.go
+++ b/pkg/catalog/config/nucleiconfig.go
@@ -46,9 +46,10 @@ type Config struct {
LatestNucleiIgnoreHash string `json:"nuclei-latest-ignore-hash,omitempty"`
// internal / unexported fields
- disableUpdates bool `json:"-"` // disable updates both version check and template updates
- homeDir string `json:"-"` // User Home Directory
- configDir string `json:"-"` // Nuclei Global Config Directory
+ disableUpdates bool `json:"-"` // disable updates both version check and template updates
+ homeDir string `json:"-"` // User Home Directory
+ configDir string `json:"-"` // Nuclei Global Config Directory
+ debugArgs []string `json:"-"` // debug args
}
// IsCustomTemplate determines whether a given template is custom-built or part of the official Nuclei templates.
@@ -329,6 +330,46 @@ func (c *Config) copyIgnoreFile() {
}
}
+// IsDebugArgEnabled checks if debug arg is enabled
+// this could be a feature specific to debugging like PPROF or printing stats
+// of max host error etc
+func (c *Config) IsDebugArgEnabled(arg string) bool {
+ for _, v := range c.debugArgs {
+ if v == arg {
+ return true
+ }
+ }
+ return false
+}
+
+// parseDebugArgs from string
+func (c *Config) parseDebugArgs(data string) {
+ // use space as seperator instead of commas
+ tmp := strings.Fields(data)
+ for _, v := range tmp {
+ key := v
+ value := ""
+ // if it is key value pair then split it
+ if strings.Contains(v, "=") {
+ parts := strings.SplitN(v, "=", 2)
+ if len(parts) != 2 {
+ continue
+ }
+ key, value = strings.TrimSpace(parts[0]), strings.TrimSpace(parts[1])
+ }
+ if value == "false" || value == "0" {
+ // if false or disabled then skip
+ continue
+ }
+ switch key {
+ case DebugArgHostErrorStats:
+ c.debugArgs = append(c.debugArgs, DebugArgHostErrorStats)
+ case DebugExportURLPattern:
+ c.debugArgs = append(c.debugArgs, DebugExportURLPattern)
+ }
+ }
+}
+
func init() {
// first attempt to migrate all files from old config directory to new config directory
goflags.AttemptConfigMigration() // regardless how many times this is called it will only migrate once based on condition
@@ -377,6 +418,7 @@ func init() {
// and even if it is changed we don't follow it since it is not expected behavior
// If custom templates are in default locations only then they are loaded while running nuclei
DefaultConfig.SetTemplatesDir(DefaultConfig.TemplatesDirectory)
+ DefaultConfig.parseDebugArgs(env.GetEnvOrDefault("NUCLEI_ARGS", ""))
}
// Add Default Config adds default when .templates-config.json file is not present
diff --git a/pkg/catalog/disk/catalog.go b/pkg/catalog/disk/catalog.go
index e37744f5b0..46f6c4693e 100644
--- a/pkg/catalog/disk/catalog.go
+++ b/pkg/catalog/disk/catalog.go
@@ -11,17 +11,25 @@ import (
// DiskCatalog is a template catalog helper implementation based on disk
type DiskCatalog struct {
templatesDirectory string
- templatesFS fs.FS // TODO: Refactor to use this
+ templatesFS fs.FS // Due to issues with how Go has implemented fs.FS, we'll have to also implement normal os operations, as well. See: https://github.com/golang/go/issues/44279
}
// NewCatalog creates a new Catalog structure using provided input items
// using disk based items
func NewCatalog(directory string) *DiskCatalog {
catalog := &DiskCatalog{templatesDirectory: directory}
- if directory != "" {
- catalog.templatesFS = os.DirFS(directory)
- } else {
- catalog.templatesFS = os.DirFS(config.DefaultConfig.GetTemplateDir())
+ if directory == "" {
+ catalog.templatesDirectory = config.DefaultConfig.GetTemplateDir()
+ }
+ return catalog
+}
+
+// NewFSCatalog creates a new Catalog structure using provided input items
+// using the fs.FS as its filesystem.
+func NewFSCatalog(fs fs.FS, directory string) *DiskCatalog {
+ catalog := &DiskCatalog{
+ templatesDirectory: directory,
+ templatesFS: fs,
}
return catalog
}
@@ -29,11 +37,15 @@ func NewCatalog(directory string) *DiskCatalog {
// OpenFile opens a file and returns an io.ReadCloser to the file.
// It is used to read template and payload files based on catalog responses.
func (d *DiskCatalog) OpenFile(filename string) (io.ReadCloser, error) {
- file, err := os.Open(filename)
- if err != nil {
- if file, errx := os.Open(BackwardsCompatiblePaths(d.templatesDirectory, filename)); errx == nil {
- return file, nil
+ if d.templatesFS == nil {
+ file, err := os.Open(filename)
+ if err != nil {
+ if file, errx := os.Open(BackwardsCompatiblePaths(d.templatesDirectory, filename)); errx == nil {
+ return file, nil
+ }
}
+ return file, err
}
- return file, err
+
+ return d.templatesFS.Open(filename)
}
diff --git a/pkg/catalog/disk/find.go b/pkg/catalog/disk/find.go
index 54c1dc27c9..76a6851b58 100644
--- a/pkg/catalog/disk/find.go
+++ b/pkg/catalog/disk/find.go
@@ -10,6 +10,15 @@ import (
"github.com/logrusorgru/aurora"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/catalog/disk/find.go
+ "github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/catalog/disk/find.go
+>>>>>>> projectdiscovery-main
stringsutil "github.com/projectdiscovery/utils/strings"
updateutils "github.com/projectdiscovery/utils/update"
urlutil "github.com/projectdiscovery/utils/url"
@@ -79,17 +88,21 @@ func (c *DiskCatalog) GetTemplatePath(target string) ([]string, error) {
}
// try to handle deprecated template paths
- absPath := BackwardsCompatiblePaths(c.templatesDirectory, target)
- if absPath != target && strings.TrimPrefix(absPath, c.templatesDirectory+string(filepath.Separator)) != target {
- if config.DefaultConfig.LogAllEvents {
- gologger.DefaultLogger.Print().Msgf("[%v] requested Template path %s is deprecated, please update to %s\n", aurora.Yellow("WRN").String(), target, absPath)
+ absPath := target
+ if c.templatesFS == nil {
+ absPath = BackwardsCompatiblePaths(c.templatesDirectory, target)
+ if absPath != target && strings.TrimPrefix(absPath, c.templatesDirectory+string(filepath.Separator)) != target {
+ if config.DefaultConfig.LogAllEvents {
+ gologger.DefaultLogger.Print().Msgf("[%v] requested Template path %s is deprecated, please update to %s\n", aurora.Yellow("WRN").String(), target, absPath)
+ }
+ deprecatedPathsCounter++
}
- deprecatedPathsCounter++
- }
- absPath, err := c.convertPathToAbsolute(absPath)
- if err != nil {
- return nil, errors.Wrapf(err, "could not find template file")
+ var err error
+ absPath, err = c.convertPathToAbsolute(absPath)
+ if err != nil {
+ return nil, errors.Wrapf(err, "could not find template file")
+ }
}
// Template input is either a file or a directory
@@ -143,24 +156,60 @@ func (c *DiskCatalog) findGlobPathMatches(absPath string, processed map[string]s
if c.templatesDirectory == "" {
templateDir = "./"
}
- matches, _ := fs.Glob(os.DirFS(filepath.Join(templateDir, "http")), inputGlob)
- if len(matches) != 0 {
+
+ if c.templatesFS == nil {
+ matches, _ := fs.Glob(os.DirFS(filepath.Join(templateDir, "http")), inputGlob)
+ if len(matches) != 0 {
+ return matches
+ }
+
+ // condition to support network cve related globs
+ matches, _ = fs.Glob(os.DirFS(filepath.Join(templateDir, "network")), inputGlob)
+ return matches
+ } else {
+ sub, err := fs.Sub(c.templatesFS, filepath.Join(templateDir, "http"))
+ if err != nil {
+ return nil
+ }
+ matches, _ := fs.Glob(sub, inputGlob)
+ if len(matches) != 0 {
+ return matches
+ }
+
+ // condition to support network cve related globs
+ sub, err = fs.Sub(c.templatesFS, filepath.Join(templateDir, "network"))
+ if err != nil {
+ return nil
+ }
+ matches, _ = fs.Glob(sub, inputGlob)
return matches
}
- // condition to support network cve related globs
- matches, _ = fs.Glob(os.DirFS(filepath.Join(templateDir, "network")), inputGlob)
- return matches
}
var matched []string
- matches, err := fs.Glob(c.templatesFS, relPath)
- if len(matches) != 0 {
- matched = append(matched, matches...)
+ var matches []string
+ if c.templatesFS == nil {
+ var err error
+ matches, err = filepath.Glob(relPath)
+ if len(matches) != 0 {
+ matched = append(matched, matches...)
+ } else {
+ matched = append(matched, OldPathsResolver(relPath)...)
+ }
+ if err != nil && len(matched) == 0 {
+ return nil, errors.Errorf("wildcard found, but unable to glob: %s\n", err)
+ }
} else {
- matched = append(matched, OldPathsResolver(relPath)...)
- }
- if err != nil && len(matched) == 0 {
- return nil, errors.Errorf("wildcard found, but unable to glob: %s\n", err)
+ var err error
+ matches, err = fs.Glob(c.templatesFS, relPath)
+ if len(matches) != 0 {
+ matched = append(matched, matches...)
+ } else {
+ matched = append(matched, OldPathsResolver(relPath)...)
+ }
+ if err != nil && len(matched) == 0 {
+ return nil, errors.Errorf("wildcard found, but unable to glob: %s\n", err)
+ }
}
results := make([]string, 0, len(matches))
for _, match := range matches {
@@ -175,11 +224,28 @@ func (c *DiskCatalog) findGlobPathMatches(absPath string, processed map[string]s
// findFileMatches finds if a path is an absolute file. If the path
// is a file, it returns true otherwise false with no errors.
func (c *DiskCatalog) findFileMatches(absPath string, processed map[string]struct{}) (match string, matched bool, err error) {
- info, err := os.Stat(absPath)
+ if c.templatesFS != nil {
+ absPath = strings.Trim(absPath, "/")
+ }
+ var info fs.File
+ if c.templatesFS == nil {
+ info, err = os.Open(absPath)
+ } else {
+ // If we were given no path, then it's not a file, it's the root, and we can quietly return.
+ if absPath == "" {
+ return "", false, nil
+ }
+
+ info, err = c.templatesFS.Open(absPath)
+ }
+ if err != nil {
+ return "", false, err
+ }
+ stat, err := info.Stat()
if err != nil {
return "", false, err
}
- if !info.Mode().IsRegular() {
+ if !stat.Mode().IsRegular() {
return "", false, nil
}
if _, ok := processed[absPath]; !ok {
@@ -192,22 +258,49 @@ func (c *DiskCatalog) findFileMatches(absPath string, processed map[string]struc
// findDirectoryMatches finds matches for templates from a directory
func (c *DiskCatalog) findDirectoryMatches(absPath string, processed map[string]struct{}) ([]string, error) {
var results []string
- err := filepath.WalkDir(
- absPath,
- func(path string, d fs.DirEntry, err error) error {
- // continue on errors
- if err != nil {
+ var err error
+ if c.templatesFS == nil {
+ err = filepath.WalkDir(
+ absPath,
+ func(path string, d fs.DirEntry, err error) error {
+ // continue on errors
+ if err != nil {
+ return nil
+ }
+ if !d.IsDir() && config.GetTemplateFormatFromExt(path) != config.Unknown {
+ if _, ok := processed[path]; !ok {
+ results = append(results, path)
+ processed[path] = struct{}{}
+ }
+ }
return nil
- }
- if !d.IsDir() && config.GetTemplateFormatFromExt(path) != config.Unknown {
- if _, ok := processed[path]; !ok {
- results = append(results, path)
- processed[path] = struct{}{}
+ },
+ )
+ } else {
+ // For the special case of the root directory, we need to pass "." to `fs.WalkDir`.
+ if absPath == "" {
+ absPath = "."
+ }
+ absPath = strings.TrimSuffix(absPath, "/")
+
+ err = fs.WalkDir(
+ c.templatesFS,
+ absPath,
+ func(path string, d fs.DirEntry, err error) error {
+ // continue on errors
+ if err != nil {
+ return nil
}
- }
- return nil
- },
- )
+ if !d.IsDir() && config.GetTemplateFormatFromExt(path) != config.Unknown {
+ if _, ok := processed[path]; !ok {
+ results = append(results, path)
+ processed[path] = struct{}{}
+ }
+ }
+ return nil
+ },
+ )
+ }
return results, err
}
diff --git a/pkg/catalog/disk/path.go b/pkg/catalog/disk/path.go
index c031295dab..3ed2b48a90 100644
--- a/pkg/catalog/disk/path.go
+++ b/pkg/catalog/disk/path.go
@@ -2,6 +2,7 @@ package disk
import (
"fmt"
+ "io/fs"
"os"
"path/filepath"
"strings"
@@ -21,6 +22,11 @@ func (c *DiskCatalog) ResolvePath(templateName, second string) (string, error) {
if filepath.IsAbs(templateName) {
return templateName, nil
}
+ if c.templatesFS != nil {
+ if potentialPath, err := c.tryResolve(templateName); err != errNoValidCombination {
+ return potentialPath, nil
+ }
+ }
if second != "" {
secondBasePath := filepath.Join(filepath.Dir(second), templateName)
if potentialPath, err := c.tryResolve(secondBasePath); err != errNoValidCombination {
@@ -28,17 +34,19 @@ func (c *DiskCatalog) ResolvePath(templateName, second string) (string, error) {
}
}
- curDirectory, err := os.Getwd()
- if err != nil {
- return "", err
- }
+ if c.templatesFS == nil {
+ curDirectory, err := os.Getwd()
+ if err != nil {
+ return "", err
+ }
- templatePath := filepath.Join(curDirectory, templateName)
- if potentialPath, err := c.tryResolve(templatePath); err != errNoValidCombination {
- return potentialPath, nil
+ templatePath := filepath.Join(curDirectory, templateName)
+ if potentialPath, err := c.tryResolve(templatePath); err != errNoValidCombination {
+ return potentialPath, nil
+ }
}
- templatePath = filepath.Join(config.DefaultConfig.GetTemplateDir(), templateName)
+ templatePath := filepath.Join(config.DefaultConfig.GetTemplateDir(), templateName)
if potentialPath, err := c.tryResolve(templatePath); err != errNoValidCombination {
return potentialPath, nil
}
@@ -50,8 +58,14 @@ var errNoValidCombination = errors.New("no valid combination found")
// tryResolve attempts to load locate the target by iterating across all the folders tree
func (c *DiskCatalog) tryResolve(fullPath string) (string, error) {
- if fileutil.FileOrFolderExists(fullPath) {
- return fullPath, nil
+ if c.templatesFS == nil {
+ if fileutil.FileOrFolderExists(fullPath) {
+ return fullPath, nil
+ }
+ } else {
+ if _, err := fs.Stat(c.templatesFS, fullPath); err == nil {
+ return fullPath, nil
+ }
}
return "", errNoValidCombination
}
diff --git a/pkg/catalog/loader/filter/path_filter.go b/pkg/catalog/loader/filter/path_filter.go
index 1a92c4c8ba..24a8d7bd20 100644
--- a/pkg/catalog/loader/filter/path_filter.go
+++ b/pkg/catalog/loader/filter/path_filter.go
@@ -1,7 +1,15 @@
package filter
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog"
+=======
+<<<<<<< HEAD:v2/pkg/catalog/loader/filter/path_filter.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/catalog/loader/filter/path_filter.go
+>>>>>>> projectdiscovery-main
)
// PathFilter is a path based template filter
diff --git a/pkg/catalog/loader/loader.go b/pkg/catalog/loader/loader.go
index f2909143e2..13e0a0a312 100644
--- a/pkg/catalog/loader/loader.go
+++ b/pkg/catalog/loader/loader.go
@@ -7,6 +7,7 @@ import (
"os"
"sort"
"strings"
+ "sync"
"github.com/Explorer1092/nuclei/v3/pkg/catalog"
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
@@ -21,8 +22,23 @@ import (
"github.com/logrusorgru/aurora"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader/filter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/keys"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/stats"
+ "github.com/projectdiscovery/nuclei/v3/pkg/workflows"
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
errorutil "github.com/projectdiscovery/utils/errors"
+ sliceutil "github.com/projectdiscovery/utils/slice"
stringsutil "github.com/projectdiscovery/utils/strings"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -30,6 +46,7 @@ import (
const (
httpPrefix = "http://"
httpsPrefix = "https://"
+ AuthStoreId = "auth_store"
)
var (
@@ -38,6 +55,7 @@ var (
// Config contains the configuration options for the loader
type Config struct {
+ StoreId string // used to set store id (optional)
Templates []string
TemplateURLs []string
Workflows []string
@@ -64,6 +82,7 @@ type Config struct {
// Store is a storage for loaded nuclei templates
type Store struct {
+ id string // id of the store (optional)
tagFilter *templates.TagFilter
pathFilter *filter.PathFilter
config *Config
@@ -129,6 +148,7 @@ func New(cfg *Config) (*Store, error) {
// Create a tag filter based on provided configuration
store := &Store{
+ id: cfg.StoreId,
config: cfg,
tagFilter: tagFilter,
pathFilter: filter.NewPathFilter(&filter.PathFilterConfig{
@@ -227,6 +247,10 @@ func (store *Store) ReadTemplateFromURI(uri string, remote bool) ([]byte, error)
}
}
+func (store *Store) ID() string {
+ return store.id
+}
+
// Templates returns all the templates in the store
func (store *Store) Templates() []*templates.Template {
return store.templates
@@ -341,7 +365,22 @@ func (store *Store) areWorkflowOrTemplatesValid(filteredTemplatePaths map[string
if err != nil {
if isParsingError("Error occurred parsing template %s: %s\n", templatePath, err) {
areTemplatesValid = false
+ continue
}
+ } else if template == nil {
+ // NOTE(dwisiswant0): possibly global matchers template.
+ // This could definitely be handled better, for example by returning an
+ // `ErrGlobalMatchersTemplate` during `templates.Parse` and checking it
+ // with `errors.Is`.
+ //
+ // However, I’m not sure if every reference to it should be handled
+ // that way. Returning a `templates.Template` pointer would mean it’s
+ // an active template (sending requests), and adding a specific field
+ // like `isGlobalMatchers` in `templates.Template` (then checking it
+ // with a `*templates.Template.IsGlobalMatchersEnabled` method) would
+ // just introduce more unknown issues - like during template
+ // clustering, AFAIK.
+ continue
} else {
if existingTemplatePath, found := templateIDPathMap[template.ID]; !found {
templateIDPathMap[template.ID] = templatePath
@@ -425,14 +464,14 @@ func (store *Store) LoadTemplatesWithTags(templatesList, tags []string) []*templ
store.logErroredTemplates(errs)
templatePathMap := store.pathFilter.Match(includedTemplates)
- loadedTemplates := make([]*templates.Template, 0, len(templatePathMap))
+ loadedTemplates := sliceutil.NewSyncSlice[*templates.Template]()
loadTemplate := func(tmpl *templates.Template) {
- loadedTemplates = append(loadedTemplates, tmpl)
+ loadedTemplates.Append(tmpl)
// increment signed/unsigned counters
if tmpl.Verified {
if tmpl.TemplateVerifier == "" {
- templates.SignatureStats[templates.PDVerifier].Add(1)
+ templates.SignatureStats[keys.PDVerifier].Add(1)
} else {
templates.SignatureStats[tmpl.TemplateVerifier].Add(1)
}
@@ -441,80 +480,101 @@ func (store *Store) LoadTemplatesWithTags(templatesList, tags []string) []*templ
}
}
+ var wgLoadTemplates sync.WaitGroup
+
for templatePath := range templatePathMap {
- loaded, err := store.config.ExecutorOptions.Parser.LoadTemplate(templatePath, store.tagFilter, tags, store.config.Catalog)
- if loaded || store.pathFilter.MatchIncluded(templatePath) {
- parsed, err := templates.Parse(templatePath, store.preprocessor, store.config.ExecutorOptions)
- if err != nil {
- // exclude templates not compatible with offline matching from total runtime warning stats
- if !errors.Is(err, templates.ErrIncompatibleWithOfflineMatching) {
- stats.Increment(templates.RuntimeWarningsStats)
- }
- gologger.Warning().Msgf("Could not parse template %s: %s\n", templatePath, err)
- } else if parsed != nil {
- if !parsed.Verified && store.config.ExecutorOptions.Options.DisableUnsignedTemplates {
- // skip unverified templates when prompted to
- stats.Increment(templates.SkippedUnsignedStats)
- continue
- }
- // if template has request signature like aws then only signed and verified templates are allowed
- if parsed.UsesRequestSignature() && !parsed.Verified {
- stats.Increment(templates.SkippedRequestSignatureStats)
- continue
- }
- // DAST only templates
- if store.config.ExecutorOptions.Options.DAST {
- // check if the template is a DAST template
- if parsed.IsFuzzing() {
- loadTemplate(parsed)
+ wgLoadTemplates.Add(1)
+ go func(templatePath string) {
+ defer wgLoadTemplates.Done()
+
+ loaded, err := store.config.ExecutorOptions.Parser.LoadTemplate(templatePath, store.tagFilter, tags, store.config.Catalog)
+ if loaded || store.pathFilter.MatchIncluded(templatePath) {
+ parsed, err := templates.Parse(templatePath, store.preprocessor, store.config.ExecutorOptions)
+ if err != nil {
+ // exclude templates not compatible with offline matching from total runtime warning stats
+ if !errors.Is(err, templates.ErrIncompatibleWithOfflineMatching) {
+ stats.Increment(templates.RuntimeWarningsStats)
}
- } else if len(parsed.RequestsHeadless) > 0 && !store.config.ExecutorOptions.Options.Headless {
- // donot include headless template in final list if headless flag is not set
- stats.Increment(templates.ExcludedHeadlessTmplStats)
- if config.DefaultConfig.LogAllEvents {
- gologger.Print().Msgf("[%v] Headless flag is required for headless template '%s'.\n", aurora.Yellow("WRN").String(), templatePath)
+ gologger.Warning().Msgf("Could not parse template %s: %s\n", templatePath, err)
+ } else if parsed != nil {
+ if !parsed.Verified && store.config.ExecutorOptions.Options.DisableUnsignedTemplates {
+ // skip unverified templates when prompted to
+ stats.Increment(templates.SkippedUnsignedStats)
+ return
}
- } else if len(parsed.RequestsCode) > 0 && !store.config.ExecutorOptions.Options.EnableCodeTemplates {
- // donot include 'Code' protocol custom template in final list if code flag is not set
- stats.Increment(templates.ExcludedCodeTmplStats)
- if config.DefaultConfig.LogAllEvents {
- gologger.Print().Msgf("[%v] Code flag is required for code protocol template '%s'.\n", aurora.Yellow("WRN").String(), templatePath)
+
+ if parsed.SelfContained && !store.config.ExecutorOptions.Options.EnableSelfContainedTemplates {
+ stats.Increment(templates.ExcludedSelfContainedStats)
+ return
}
- } else if len(parsed.RequestsCode) > 0 && !parsed.Verified && len(parsed.Workflows) == 0 {
- // donot include unverified 'Code' protocol custom template in final list
- stats.Increment(templates.SkippedCodeTmplTamperedStats)
- // these will be skipped so increment skip counter
- stats.Increment(templates.SkippedUnsignedStats)
- if config.DefaultConfig.LogAllEvents {
- gologger.Print().Msgf("[%v] Tampered/Unsigned template at %v.\n", aurora.Yellow("WRN").String(), templatePath)
+
+ if parsed.HasFileProtocol() && !store.config.ExecutorOptions.Options.EnableFileTemplates {
+ stats.Increment(templates.ExcludedFileStats)
+ return
}
- } else if parsed.IsFuzzing() && !store.config.ExecutorOptions.Options.DAST {
- stats.Increment(templates.ExludedDastTmplStats)
- if config.DefaultConfig.LogAllEvents {
- gologger.Print().Msgf("[%v] -dast flag is required for DAST template '%s'.\n", aurora.Yellow("WRN").String(), templatePath)
+
+ // if template has request signature like aws then only signed and verified templates are allowed
+ if parsed.UsesRequestSignature() && !parsed.Verified {
+ stats.Increment(templates.SkippedRequestSignatureStats)
+ return
+ }
+ // DAST only templates
+ // Skip DAST filter when loading auth templates
+ if store.ID() != AuthStoreId && store.config.ExecutorOptions.Options.DAST {
+ // check if the template is a DAST template
+ if parsed.IsFuzzing() {
+ loadTemplate(parsed)
+ }
+ } else if len(parsed.RequestsHeadless) > 0 && !store.config.ExecutorOptions.Options.Headless {
+ // donot include headless template in final list if headless flag is not set
+ stats.Increment(templates.ExcludedHeadlessTmplStats)
+ if config.DefaultConfig.LogAllEvents {
+ gologger.Print().Msgf("[%v] Headless flag is required for headless template '%s'.\n", aurora.Yellow("WRN").String(), templatePath)
+ }
+ } else if len(parsed.RequestsCode) > 0 && !store.config.ExecutorOptions.Options.EnableCodeTemplates {
+ // donot include 'Code' protocol custom template in final list if code flag is not set
+ stats.Increment(templates.ExcludedCodeTmplStats)
+ if config.DefaultConfig.LogAllEvents {
+ gologger.Print().Msgf("[%v] Code flag is required for code protocol template '%s'.\n", aurora.Yellow("WRN").String(), templatePath)
+ }
+ } else if len(parsed.RequestsCode) > 0 && !parsed.Verified && len(parsed.Workflows) == 0 {
+ // donot include unverified 'Code' protocol custom template in final list
+ stats.Increment(templates.SkippedCodeTmplTamperedStats)
+ // these will be skipped so increment skip counter
+ stats.Increment(templates.SkippedUnsignedStats)
+ if config.DefaultConfig.LogAllEvents {
+ gologger.Print().Msgf("[%v] Tampered/Unsigned template at %v.\n", aurora.Yellow("WRN").String(), templatePath)
+ }
+ } else if parsed.IsFuzzing() && !store.config.ExecutorOptions.Options.DAST {
+ stats.Increment(templates.ExludedDastTmplStats)
+ if config.DefaultConfig.LogAllEvents {
+ gologger.Print().Msgf("[%v] -dast flag is required for DAST template '%s'.\n", aurora.Yellow("WRN").String(), templatePath)
+ }
+ } else {
+ loadTemplate(parsed)
}
- } else {
- loadTemplate(parsed)
}
}
- }
- if err != nil {
- if strings.Contains(err.Error(), templates.ErrExcluded.Error()) {
- stats.Increment(templates.TemplatesExcludedStats)
- if config.DefaultConfig.LogAllEvents {
- gologger.Print().Msgf("[%v] %v\n", aurora.Yellow("WRN").String(), err.Error())
+ if err != nil {
+ if strings.Contains(err.Error(), templates.ErrExcluded.Error()) {
+ stats.Increment(templates.TemplatesExcludedStats)
+ if config.DefaultConfig.LogAllEvents {
+ gologger.Print().Msgf("[%v] %v\n", aurora.Yellow("WRN").String(), err.Error())
+ }
+ return
}
- continue
+ gologger.Warning().Msg(err.Error())
}
- gologger.Warning().Msg(err.Error())
- }
+ }(templatePath)
}
- sort.SliceStable(loadedTemplates, func(i, j int) bool {
- return loadedTemplates[i].Path < loadedTemplates[j].Path
+ wgLoadTemplates.Wait()
+
+ sort.SliceStable(loadedTemplates.Slice, func(i, j int) bool {
+ return loadedTemplates.Slice[i].Path < loadedTemplates.Slice[j].Path
})
- return loadedTemplates
+ return loadedTemplates.Slice
}
// IsHTTPBasedProtocolUsed returns true if http/headless protocol is being used for
diff --git a/pkg/catalog/loader/loader_test.go b/pkg/catalog/loader/loader_test.go
index f6da9767dc..ef0c9429e3 100644
--- a/pkg/catalog/loader/loader_test.go
+++ b/pkg/catalog/loader/loader_test.go
@@ -4,8 +4,18 @@ import (
"reflect"
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/catalog/disk"
+=======
+<<<<<<< HEAD:v2/pkg/catalog/loader/loader_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/disk"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/catalog/loader/loader_test.go
+>>>>>>> projectdiscovery-main
"github.com/stretchr/testify/require"
)
diff --git a/pkg/catalog/loader/remote_loader.go b/pkg/catalog/loader/remote_loader.go
index 263967c7ac..f8490cd9e3 100644
--- a/pkg/catalog/loader/remote_loader.go
+++ b/pkg/catalog/loader/remote_loader.go
@@ -8,8 +8,18 @@ import (
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/templates/extensions"
"github.com/Explorer1092/nuclei/v3/pkg/utils"
+=======
+<<<<<<< HEAD:v2/pkg/catalog/loader/remote_loader.go
+ "github.com/Explorer1092/nuclei/v2/pkg/templates/extensions"
+ "github.com/Explorer1092/nuclei/v2/pkg/utils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/extensions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/catalog/loader/remote_loader.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
stringsutil "github.com/projectdiscovery/utils/strings"
)
diff --git a/pkg/core/engine.go b/pkg/core/engine.go
index 6ae31ece00..b61cb39cce 100644
--- a/pkg/core/engine.go
+++ b/pkg/core/engine.go
@@ -1,9 +1,22 @@
package core
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/core/engine.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/core/engine.go
+>>>>>>> projectdiscovery-main
)
// Engine is an executer for running Nuclei Templates/Workflows.
diff --git a/pkg/core/executors.go b/pkg/core/executors.go
index cb41ddf250..ed7015e277 100644
--- a/pkg/core/executors.go
+++ b/pkg/core/executors.go
@@ -12,7 +12,26 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/templates/types"
generalTypes "github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
syncutil "github.com/projectdiscovery/utils/sync"
+=======
+<<<<<<< HEAD:v2/pkg/core/executors.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+ generalTypes "github.com/Explorer1092/nuclei/v2/pkg/types"
+ "github.com/remeh/sizedwaitgroup"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ generalTypes "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ syncutil "github.com/projectdiscovery/utils/sync"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/core/executors.go
+>>>>>>> projectdiscovery-main
)
// Executors are low level executors that deals with template execution on a target
@@ -27,7 +46,7 @@ func (e *Engine) executeAllSelfContained(ctx context.Context, alltemplates []*te
var match bool
ctx := scan.NewScanContext(ctx, contextargs.New(ctx))
if e.Callback != nil {
- if results, err := template.Executer.ExecuteWithResults(ctx); err != nil {
+ if results, err := template.Executer.ExecuteWithResults(ctx); err == nil {
for _, result := range results {
e.Callback(result)
}
@@ -107,7 +126,7 @@ func (e *Engine) executeTemplateWithTargets(ctx context.Context, template *templ
currentInfo.Unlock()
// Skip if the host has had errors
- if e.executerOpts.HostErrorsCache != nil && e.executerOpts.HostErrorsCache.Check(scannedValue.ID()) {
+ if e.executerOpts.HostErrorsCache != nil && e.executerOpts.HostErrorsCache.Check(e.executerOpts.ProtocolType.String(), contextargs.NewWithMetaInput(ctx, scannedValue)) {
return true
}
@@ -129,7 +148,7 @@ func (e *Engine) executeTemplateWithTargets(ctx context.Context, template *templ
match = e.executeWorkflow(ctx, template.CompiledWorkflow)
default:
if e.Callback != nil {
- if results, err := template.Executer.ExecuteWithResults(ctx); err != nil {
+ if results, err := template.Executer.ExecuteWithResults(ctx); err == nil {
for _, result := range results {
e.Callback(result)
}
@@ -194,7 +213,7 @@ func (e *Engine) executeTemplatesOnTarget(ctx context.Context, alltemplates []*t
match = e.executeWorkflow(ctx, template.CompiledWorkflow)
default:
if e.Callback != nil {
- if results, err := template.Executer.ExecuteWithResults(ctx); err != nil {
+ if results, err := template.Executer.ExecuteWithResults(ctx); err == nil {
for _, result := range results {
e.Callback(result)
}
diff --git a/pkg/core/workflow_execute.go b/pkg/core/workflow_execute.go
index 9375d04cc5..4f7a21b61f 100644
--- a/pkg/core/workflow_execute.go
+++ b/pkg/core/workflow_execute.go
@@ -10,7 +10,21 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/scan"
"github.com/Explorer1092/nuclei/v3/pkg/workflows"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
syncutil "github.com/projectdiscovery/utils/sync"
+=======
+<<<<<<< HEAD:v2/pkg/core/workflow_execute.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/workflows"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/workflows"
+ syncutil "github.com/projectdiscovery/utils/sync"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/core/workflow_execute.go
+>>>>>>> projectdiscovery-main
)
const workflowStepExecutionError = "[%s] Could not execute workflow step: %s\n"
@@ -98,7 +112,7 @@ func (e *Engine) runWorkflowStep(template *workflows.WorkflowTemplate, ctx *scan
}
if err != nil {
if w.Options.HostErrorsCache != nil {
- w.Options.HostErrorsCache.MarkFailed(ctx.Input.MetaInput.ID(), err)
+ w.Options.HostErrorsCache.MarkFailed(w.Options.ProtocolType.String(), ctx.Input, err)
}
if len(template.Executers) == 1 {
mainErr = err
@@ -139,7 +153,8 @@ func (e *Engine) runWorkflowStep(template *workflows.WorkflowTemplate, ctx *scan
defer swg.Done()
// create a new context with the same input but with unset callbacks
- subCtx := scan.NewScanContext(ctx.Context(), ctx.Input)
+ // clone the Input so that other parallel executions won't overwrite the shared variables when subsequent templates are running
+ subCtx := scan.NewScanContext(ctx.Context(), ctx.Input.Clone())
if err := e.runWorkflowStep(subtemplate, subCtx, results, swg, w); err != nil {
gologger.Warning().Msgf(workflowStepExecutionError, subtemplate.Template, err)
}
diff --git a/pkg/core/workflow_execute_test.go b/pkg/core/workflow_execute_test.go
index 1b3bd246fa..e0fad7f167 100644
--- a/pkg/core/workflow_execute_test.go
+++ b/pkg/core/workflow_execute_test.go
@@ -4,6 +4,7 @@ import (
"context"
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/output"
@@ -13,6 +14,28 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/scan"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/Explorer1092/nuclei/v3/pkg/workflows"
+=======
+<<<<<<< HEAD:v2/pkg/core/workflow_execute_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/progress"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+ "github.com/Explorer1092/nuclei/v2/pkg/workflows"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/progress"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/workflows"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/core/workflow_execute_test.go
+>>>>>>> projectdiscovery-main
"github.com/stretchr/testify/require"
)
diff --git a/pkg/external/customtemplates/azure_blob.go b/pkg/external/customtemplates/azure_blob.go
index 78b3eb32aa..6b1009c31b 100644
--- a/pkg/external/customtemplates/azure_blob.go
+++ b/pkg/external/customtemplates/azure_blob.go
@@ -12,6 +12,16 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/external/customtemplates/azure_blob.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/external/customtemplates/azure_blob.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
)
diff --git a/pkg/external/customtemplates/github.go b/pkg/external/customtemplates/github.go
index 197f16a061..fabf21ef53 100644
--- a/pkg/external/customtemplates/github.go
+++ b/pkg/external/customtemplates/github.go
@@ -13,6 +13,16 @@ import (
"github.com/google/go-github/github"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/external/customtemplates/github.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/external/customtemplates/github.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
folderutil "github.com/projectdiscovery/utils/folder"
"golang.org/x/oauth2"
@@ -137,33 +147,59 @@ getRepo:
// download the git repo to a given path
func (ctr *customTemplateGitHubRepo) cloneRepo(clonePath, githubToken string) error {
- r, err := git.PlainClone(clonePath, false, &git.CloneOptions{
- URL: ctr.gitCloneURL,
- Auth: getAuth(ctr.owner, githubToken),
- })
+ cloneOpts := &git.CloneOptions{
+ URL: ctr.gitCloneURL,
+ Auth: getAuth(ctr.owner, githubToken),
+ SingleBranch: true,
+ Depth: 1,
+ }
+
+ err := cloneOpts.Validate()
+ if err != nil {
+ return err
+ }
+
+ r, err := git.PlainClone(clonePath, false, cloneOpts)
if err != nil {
return errors.Errorf("%s/%s: %s", ctr.owner, ctr.reponame, err.Error())
}
+
// Add the user as well in the config. By default, user is not set
config, _ := r.Storer.Config()
config.User.Name = ctr.owner
+
return r.SetConfig(config)
}
// performs the git pull on given repo
func (ctr *customTemplateGitHubRepo) pullChanges(repoPath, githubToken string) error {
+ pullOpts := &git.PullOptions{
+ RemoteName: "origin",
+ Auth: getAuth(ctr.owner, githubToken),
+ SingleBranch: true,
+ Depth: 1,
+ }
+
+ err := pullOpts.Validate()
+ if err != nil {
+ return err
+ }
+
r, err := git.PlainOpen(repoPath)
if err != nil {
return err
}
+
w, err := r.Worktree()
if err != nil {
return err
}
- err = w.Pull(&git.PullOptions{RemoteName: "origin", Auth: getAuth(ctr.owner, githubToken)})
+
+ err = w.Pull(pullOpts)
if err != nil {
return errors.Errorf("%s/%s: %s", ctr.owner, ctr.reponame, err.Error())
}
+
return nil
}
diff --git a/pkg/external/customtemplates/github_test.go b/pkg/external/customtemplates/github_test.go
index 03ebfe45e3..c951642618 100644
--- a/pkg/external/customtemplates/github_test.go
+++ b/pkg/external/customtemplates/github_test.go
@@ -9,6 +9,16 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/external/customtemplates/github_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/external/customtemplates/github_test.go
+>>>>>>> projectdiscovery-main
"github.com/stretchr/testify/require"
)
@@ -22,14 +32,12 @@ func TestDownloadCustomTemplatesFromGitHub(t *testing.T) {
config.DefaultConfig.SetTemplatesDir(templatesDirectory)
options := testutils.DefaultOptions
- options.GitHubTemplateRepo = []string{"projectdiscovery/nuclei-templates", "ehsandeep/nuclei-templates"}
- options.GitHubToken = os.Getenv("GITHUB_TOKEN")
+ options.GitHubTemplateRepo = []string{"projectdiscovery/nuclei-templates-test"}
ctm, err := NewCustomTemplatesManager(options)
require.Nil(t, err, "could not create custom templates manager")
ctm.Download(context.Background())
- require.DirExists(t, filepath.Join(templatesDirectory, "github", "projectdiscovery", "nuclei-templates"), "cloned directory does not exists")
- require.DirExists(t, filepath.Join(templatesDirectory, "github", "ehsandeep", "nuclei-templates"), "cloned directory does not exists")
+ require.DirExists(t, filepath.Join(templatesDirectory, "github", "projectdiscovery", "nuclei-templates-test"), "cloned directory does not exists")
}
diff --git a/pkg/external/customtemplates/gitlab.go b/pkg/external/customtemplates/gitlab.go
index 1ac2b38def..fddb09324a 100644
--- a/pkg/external/customtemplates/gitlab.go
+++ b/pkg/external/customtemplates/gitlab.go
@@ -9,6 +9,16 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/external/customtemplates/gitlab.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/external/customtemplates/gitlab.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
"github.com/xanzy/go-gitlab"
)
@@ -83,8 +93,8 @@ func (bk *customTemplateGitLabRepo) Download(_ context.Context) {
// Get the directory listing for the files in the project
tree, _, err := bk.gitLabClient.Repositories.ListTree(projectID, &gitlab.ListTreeOptions{
- Ref: gitlab.String(project.DefaultBranch),
- Recursive: gitlab.Bool(true),
+ Ref: gitlab.Ptr(project.DefaultBranch),
+ Recursive: gitlab.Ptr(true),
})
if err != nil {
gologger.Error().Msgf("error retrieving files from GitLab project: %s (%d) %s", project.Name, projectID, err)
@@ -95,7 +105,7 @@ func (bk *customTemplateGitLabRepo) Download(_ context.Context) {
// If the object is not a file or file extension is not .yaml, skip it
if file.Type == "blob" && filepath.Ext(file.Path) == ".yaml" {
gf := &gitlab.GetFileOptions{
- Ref: gitlab.String(project.DefaultBranch),
+ Ref: gitlab.Ptr(project.DefaultBranch),
}
f, _, err := bk.gitLabClient.RepositoryFiles.GetFile(projectID, file.Path, gf)
if err != nil {
diff --git a/pkg/external/customtemplates/s3.go b/pkg/external/customtemplates/s3.go
index fc48e49685..159167bbbc 100644
--- a/pkg/external/customtemplates/s3.go
+++ b/pkg/external/customtemplates/s3.go
@@ -14,6 +14,16 @@ import (
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/external/customtemplates/s3.go
+ nucleiConfig "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ nucleiConfig "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/external/customtemplates/s3.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
diff --git a/pkg/external/customtemplates/templates_provider.go b/pkg/external/customtemplates/templates_provider.go
index b38783baae..9b3e95da73 100644
--- a/pkg/external/customtemplates/templates_provider.go
+++ b/pkg/external/customtemplates/templates_provider.go
@@ -3,7 +3,15 @@ package customtemplates
import (
"context"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/external/customtemplates/templates_provider.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/external/customtemplates/templates_provider.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
)
diff --git a/pkg/fuzz/analyzers/analyzers.go b/pkg/fuzz/analyzers/analyzers.go
new file mode 100644
index 0000000000..8eedb6b71b
--- /dev/null
+++ b/pkg/fuzz/analyzers/analyzers.go
@@ -0,0 +1,103 @@
+package analyzers
+
+import (
+ "math/rand"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz"
+ "github.com/projectdiscovery/retryablehttp-go"
+)
+
+// Analyzer is an interface for all the analyzers
+// that can be used for the fuzzer
+type Analyzer interface {
+ // Name returns the name of the analyzer
+ Name() string
+ // ApplyTransformation applies the transformation to the initial payload.
+ ApplyInitialTransformation(data string, params map[string]interface{}) string
+ // Analyze is the main function for the analyzer
+ Analyze(options *Options) (bool, string, error)
+}
+
+// AnalyzerTemplate is the template for the analyzer
+type AnalyzerTemplate struct {
+ // description: |
+ // Name is the name of the analyzer to use
+ // values:
+ // - time_delay
+ Name string `json:"name" yaml:"name"`
+ // description: |
+ // Parameters is the parameters for the analyzer
+ //
+ // Parameters are different for each analyzer. For example, you can customize
+ // time_delay analyzer with sleep_duration, time_slope_error_range, etc. Refer
+ // to the docs for each analyzer to get an idea about parameters.
+ Parameters map[string]interface{} `json:"parameters" yaml:"parameters"`
+}
+
+var (
+ analyzers map[string]Analyzer
+)
+
+// RegisterAnalyzer registers a new analyzer
+func RegisterAnalyzer(name string, analyzer Analyzer) {
+ analyzers[name] = analyzer
+}
+
+// GetAnalyzer returns the analyzer for a given name
+func GetAnalyzer(name string) Analyzer {
+ return analyzers[name]
+}
+
+func init() {
+ analyzers = make(map[string]Analyzer)
+}
+
+// Options contains the options for the analyzer
+type Options struct {
+ FuzzGenerated fuzz.GeneratedRequest
+ HttpClient *retryablehttp.Client
+ ResponseTimeDelay time.Duration
+ AnalyzerParameters map[string]interface{}
+}
+
+var (
+ random = rand.New(rand.NewSource(time.Now().UnixNano()))
+)
+
+// ApplyPayloadTransformations applies the payload transformations to the payload
+// It supports the below payloads -
+// - [RANDNUM] => random number between 1000 and 9999
+// - [RANDSTR] => random string of 4 characters
+func ApplyPayloadTransformations(value string) string {
+ randomInt := GetRandomInteger()
+ randomStr := randStringBytesMask(4)
+
+ value = strings.ReplaceAll(value, "[RANDNUM]", strconv.Itoa(randomInt))
+ value = strings.ReplaceAll(value, "[RANDSTR]", randomStr)
+ return value
+}
+
+const letterBytes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
+const (
+ letterIdxBits = 6 // 6 bits to represent a letter index
+ letterIdxMask = 1< sleep_duration
+// - [INFERENCE] => Inference payload for time delay analyzer
+//
+// It also applies the payload transformations to the payload
+// which includes [RANDNUM] and [RANDSTR]
+func (a *Analyzer) ApplyInitialTransformation(data string, params map[string]interface{}) string {
+ duration := DefaultSleepDuration
+ if len(params) > 0 {
+ if v, ok := params["sleep_duration"]; ok {
+ duration, ok = v.(int)
+ if !ok {
+ duration = DefaultSleepDuration
+ gologger.Warning().Msgf("Invalid sleep_duration parameter type, using default value: %d", duration)
+ }
+ }
+ }
+ data = strings.ReplaceAll(data, "[SLEEPTIME]", strconv.Itoa(duration))
+ data = analyzers.ApplyPayloadTransformations(data)
+
+ // Also support [INFERENCE] for the time delay analyzer
+ if strings.Contains(data, "[INFERENCE]") {
+ randInt := analyzers.GetRandomInteger()
+ data = strings.ReplaceAll(data, "[INFERENCE]", fmt.Sprintf("%d=%d", randInt, randInt))
+ }
+ return data
+}
+
+func (a *Analyzer) parseAnalyzerParameters(params map[string]interface{}) (int, int, float64, float64, error) {
+ requestsLimit := DefaultRequestsLimit
+ sleepDuration := DefaultSleepDuration
+ timeCorrelationErrorRange := DefaultTimeCorrelationErrorRange
+ timeSlopeErrorRange := DefaultTimeSlopeErrorRange
+
+ if len(params) == 0 {
+ return requestsLimit, sleepDuration, timeCorrelationErrorRange, timeSlopeErrorRange, nil
+ }
+ var ok bool
+ for k, v := range params {
+ switch k {
+ case "sleep_duration":
+ sleepDuration, ok = v.(int)
+ case "requests_limit":
+ requestsLimit, ok = v.(int)
+ case "time_correlation_error_range":
+ timeCorrelationErrorRange, ok = v.(float64)
+ case "time_slope_error_range":
+ timeSlopeErrorRange, ok = v.(float64)
+ }
+ if !ok {
+ return 0, 0, 0, 0, errors.Errorf("invalid parameter type for %s", k)
+ }
+ }
+ return requestsLimit, sleepDuration, timeCorrelationErrorRange, timeSlopeErrorRange, nil
+}
+
+// Analyze is the main function for the analyzer
+func (a *Analyzer) Analyze(options *analyzers.Options) (bool, string, error) {
+ if options.ResponseTimeDelay < defaultSleepTimeDuration {
+ return false, "", nil
+ }
+
+ // Parse parameters for this analyzer if any or use default values
+ requestsLimit, sleepDuration, timeCorrelationErrorRange, timeSlopeErrorRange, err :=
+ a.parseAnalyzerParameters(options.AnalyzerParameters)
+ if err != nil {
+ return false, "", err
+ }
+
+ reqSender := func(delay int) (float64, error) {
+ gr := options.FuzzGenerated
+ replaced := strings.ReplaceAll(gr.OriginalPayload, "[SLEEPTIME]", strconv.Itoa(delay))
+ replaced = a.ApplyInitialTransformation(replaced, options.AnalyzerParameters)
+
+ if err := gr.Component.SetValue(gr.Key, replaced); err != nil {
+ return 0, errors.Wrap(err, "could not set value in component")
+ }
+
+ rebuilt, err := gr.Component.Rebuild()
+ if err != nil {
+ return 0, errors.Wrap(err, "could not rebuild request")
+ }
+ gologger.Verbose().Msgf("[%s] Sending request with %d delay for: %s", a.Name(), delay, rebuilt.URL.String())
+
+ timeTaken, err := doHTTPRequestWithTimeTracing(rebuilt, options.HttpClient)
+ if err != nil {
+ return 0, errors.Wrap(err, "could not do request with time tracing")
+ }
+ return timeTaken, nil
+ }
+ matched, matchReason, err := checkTimingDependency(
+ requestsLimit,
+ sleepDuration,
+ timeCorrelationErrorRange,
+ timeSlopeErrorRange,
+ reqSender,
+ )
+ if err != nil {
+ return false, "", err
+ }
+ if matched {
+ return true, matchReason, nil
+ }
+ return false, "", nil
+}
+
+// doHTTPRequestWithTimeTracing does a http request with time tracing
+func doHTTPRequestWithTimeTracing(req *retryablehttp.Request, httpclient *retryablehttp.Client) (float64, error) {
+ var ttfb time.Duration
+ var start time.Time
+
+ trace := &httptrace.ClientTrace{
+ GotFirstResponseByte: func() { ttfb = time.Since(start) },
+ }
+ req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
+ start = time.Now()
+ resp, err := httpclient.Do(req)
+ if err != nil {
+ return 0, errors.Wrap(err, "could not do request")
+ }
+
+ _, err = io.ReadAll(resp.Body)
+ if err != nil {
+ return 0, errors.Wrap(err, "could not read response body")
+ }
+ return ttfb.Seconds(), nil
+}
diff --git a/pkg/fuzz/analyzers/time/time_delay.go b/pkg/fuzz/analyzers/time/time_delay.go
new file mode 100644
index 0000000000..d3b684ceff
--- /dev/null
+++ b/pkg/fuzz/analyzers/time/time_delay.go
@@ -0,0 +1,171 @@
+// Package time implements a time delay analyzer using linear
+// regression heuristics inspired from ZAP to discover time
+// based issues.
+//
+// The approach is the one used in ZAP for timing based checks.
+// Advantages of this approach are many compared to the old approach of
+// heuristics of sleep time.
+//
+// As we are building a statistical model, we can predict if the delay
+// is random or not very quickly. Also, the payloads are alternated to send
+// a very high sleep and a very low sleep. This way the comparison is
+// faster to eliminate negative cases. Only legitimate cases are sent for
+// more verification.
+//
+// For more details on the algorithm, follow the links below:
+// - https://groups.google.com/g/zaproxy-develop/c/KGSkNHlLtqk
+// - https://github.com/zaproxy/zap-extensions/pull/5053
+//
+// This file has been implemented from its original version. It was originally licensed under the Apache License 2.0 (see LICENSE file for details).
+// The original algorithm is implemented in ZAP Active Scanner.
+package time
+
+import (
+ "errors"
+ "fmt"
+ "math"
+)
+
+type timeDelayRequestSender func(delay int) (float64, error)
+
+// checkTimingDependency checks the timing dependency for a given request
+//
+// It alternates and sends first a high request, then a low request. Each time
+// it checks if the delay of the application can be predictably controlled.
+func checkTimingDependency(
+ requestsLimit int,
+ highSleepTimeSeconds int,
+ correlationErrorRange float64,
+ slopeErrorRange float64,
+ requestSender timeDelayRequestSender,
+) (bool, string, error) {
+ if requestsLimit < 2 {
+ return false, "", errors.New("requests limit should be at least 2")
+ }
+
+ regression := newSimpleLinearRegression()
+ requestsLeft := requestsLimit
+
+ for {
+ if requestsLeft <= 0 {
+ break
+ }
+
+ isCorrelationPossible, err := sendRequestAndTestConfidence(regression, highSleepTimeSeconds, requestSender)
+ if err != nil {
+ return false, "", err
+ }
+ if !isCorrelationPossible {
+ return false, "", nil
+ }
+
+ isCorrelationPossible, err = sendRequestAndTestConfidence(regression, 1, requestSender)
+ if err != nil {
+ return false, "", err
+ }
+ if !isCorrelationPossible {
+ return false, "", nil
+ }
+ requestsLeft = requestsLeft - 2
+ }
+
+ result := regression.IsWithinConfidence(correlationErrorRange, 1.0, slopeErrorRange)
+ if result {
+ resultReason := fmt.Sprintf(
+ "[time_delay] made %d requests successfully, with a regression slope of %.2f and correlation %.2f",
+ requestsLimit,
+ regression.slope,
+ regression.correlation,
+ )
+ return result, resultReason, nil
+ }
+ return result, "", nil
+}
+
+// sendRequestAndTestConfidence sends a request and tests the confidence of delay
+func sendRequestAndTestConfidence(
+ regression *simpleLinearRegression,
+ delay int,
+ requestSender timeDelayRequestSender,
+) (bool, error) {
+ delayReceived, err := requestSender(delay)
+ if err != nil {
+ return false, err
+ }
+
+ if delayReceived < float64(delay) {
+ return false, nil
+ }
+
+ regression.AddPoint(float64(delay), delayReceived)
+
+ if !regression.IsWithinConfidence(0.3, 1.0, 0.5) {
+ return false, nil
+ }
+ return true, nil
+}
+
+// simpleLinearRegression is a simple linear regression model that can be updated at runtime.
+// It is based on the same algorithm in ZAP for doing timing checks.
+type simpleLinearRegression struct {
+ count float64
+ independentSum float64
+ dependentSum float64
+
+ // Variances
+ independentVarianceN float64
+ dependentVarianceN float64
+ sampleCovarianceN float64
+
+ slope float64
+ intercept float64
+ correlation float64
+}
+
+func newSimpleLinearRegression() *simpleLinearRegression {
+ return &simpleLinearRegression{
+ slope: 1,
+ correlation: 1,
+ }
+}
+
+func (o *simpleLinearRegression) AddPoint(x, y float64) {
+ independentResidualAdjustment := x - o.independentSum/o.count
+ dependentResidualAdjustment := y - o.dependentSum/o.count
+
+ o.count += 1
+ o.independentSum += x
+ o.dependentSum += y
+
+ if math.IsNaN(independentResidualAdjustment) {
+ return
+ }
+
+ independentResidual := x - o.independentSum/o.count
+ dependentResidual := y - o.dependentSum/o.count
+
+ o.independentVarianceN += independentResidual * independentResidualAdjustment
+ o.dependentVarianceN += dependentResidual * dependentResidualAdjustment
+ o.sampleCovarianceN += independentResidual * dependentResidualAdjustment
+
+ o.slope = o.sampleCovarianceN / o.independentVarianceN
+ o.correlation = o.slope * math.Sqrt(o.independentVarianceN/o.dependentVarianceN)
+ o.correlation *= o.correlation
+
+ // NOTE: zap had the reverse formula, changed it to the correct one
+ // for intercept. Verify if this is correct.
+ o.intercept = o.dependentSum/o.count - o.slope*(o.independentSum/o.count)
+ if math.IsNaN(o.correlation) {
+ o.correlation = 1
+ }
+}
+
+func (o *simpleLinearRegression) Predict(x float64) float64 {
+ return o.slope*x + o.intercept
+}
+
+func (o *simpleLinearRegression) IsWithinConfidence(correlationErrorRange float64, expectedSlope float64, slopeErrorRange float64,
+) bool {
+ return o.correlation > 1.0-correlationErrorRange &&
+ math.Abs(expectedSlope-o.slope) < slopeErrorRange
+}
diff --git a/pkg/fuzz/analyzers/time/time_delay_test.go b/pkg/fuzz/analyzers/time/time_delay_test.go
new file mode 100644
index 0000000000..8a71243595
--- /dev/null
+++ b/pkg/fuzz/analyzers/time/time_delay_test.go
@@ -0,0 +1,143 @@
+// Tests ported from ZAP Java version of the algorithm
+
+package time
+
+import (
+ "math"
+ "math/rand"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+)
+
+const (
+ correlationErrorRange = float64(0.1)
+ slopeErrorRange = float64(0.2)
+)
+
+var rng = rand.New(rand.NewSource(time.Now().UnixNano()))
+
+func Test_should_generate_alternating_sequences(t *testing.T) {
+ var generatedDelays []float64
+ reqSender := func(delay int) (float64, error) {
+ generatedDelays = append(generatedDelays, float64(delay))
+ return float64(delay), nil
+ }
+ matched, _, err := checkTimingDependency(4, 15, correlationErrorRange, slopeErrorRange, reqSender)
+ require.NoError(t, err)
+ require.True(t, matched)
+ require.EqualValues(t, []float64{15, 1, 15, 1}, generatedDelays)
+}
+
+func Test_should_giveup_non_injectable(t *testing.T) {
+ var timesCalled int
+ reqSender := func(delay int) (float64, error) {
+ timesCalled++
+ return 0.5, nil
+ }
+ matched, _, err := checkTimingDependency(4, 15, correlationErrorRange, slopeErrorRange, reqSender)
+ require.NoError(t, err)
+ require.False(t, matched)
+ require.Equal(t, 1, timesCalled)
+}
+
+func Test_should_giveup_slow_non_injectable(t *testing.T) {
+ var timesCalled int
+ reqSender := func(delay int) (float64, error) {
+ timesCalled++
+ return 10 + rng.Float64()*0.5, nil
+ }
+ matched, _, err := checkTimingDependency(4, 15, correlationErrorRange, slopeErrorRange, reqSender)
+ require.NoError(t, err)
+ require.False(t, matched)
+ require.LessOrEqual(t, timesCalled, 3)
+}
+
+func Test_should_giveup_slow_non_injectable_realworld(t *testing.T) {
+ var timesCalled int
+ var iteration = 0
+ counts := []float64{21, 11, 21, 11}
+ reqSender := func(delay int) (float64, error) {
+ timesCalled++
+ iteration++
+ return counts[iteration-1], nil
+ }
+ matched, _, err := checkTimingDependency(4, 15, correlationErrorRange, slopeErrorRange, reqSender)
+ require.NoError(t, err)
+ require.False(t, matched)
+ require.LessOrEqual(t, timesCalled, 4)
+}
+
+func Test_should_detect_dependence_with_small_error(t *testing.T) {
+ reqSender := func(delay int) (float64, error) {
+ return float64(delay) + rng.Float64()*0.5, nil
+ }
+ matched, reason, err := checkTimingDependency(4, 15, correlationErrorRange, slopeErrorRange, reqSender)
+ require.NoError(t, err)
+ require.True(t, matched)
+ require.NotEmpty(t, reason)
+}
+
+func Test_LinearRegression_Numerical_stability(t *testing.T) {
+ variables := [][]float64{
+ {1, 1}, {2, 2}, {3, 3}, {4, 4}, {5, 5}, {1, 1}, {2, 2}, {2, 2}, {2, 2},
+ }
+ slope := float64(1)
+ correlation := float64(1)
+
+ regression := newSimpleLinearRegression()
+ for _, v := range variables {
+ regression.AddPoint(v[0], v[1])
+ }
+ require.True(t, almostEqual(regression.slope, slope))
+ require.True(t, almostEqual(regression.correlation, correlation))
+}
+
+func Test_LinearRegression_exact_verify(t *testing.T) {
+ variables := [][]float64{
+ {1, 1}, {2, 3},
+ }
+ slope := float64(2)
+ correlation := float64(1)
+
+ regression := newSimpleLinearRegression()
+ for _, v := range variables {
+ regression.AddPoint(v[0], v[1])
+ }
+ require.True(t, almostEqual(regression.slope, slope))
+ require.True(t, almostEqual(regression.correlation, correlation))
+}
+
+func Test_LinearRegression_known_verify(t *testing.T) {
+ variables := [][]float64{
+ {1, 1.348520581}, {2, 2.524046187}, {3, 3.276944688}, {4, 4.735374498}, {5, 5.150291657},
+ }
+ slope := float64(0.981487046)
+ correlation := float64(0.979228906)
+
+ regression := newSimpleLinearRegression()
+ for _, v := range variables {
+ regression.AddPoint(v[0], v[1])
+ }
+ require.True(t, almostEqual(regression.slope, slope))
+ require.True(t, almostEqual(regression.correlation, correlation))
+}
+
+func Test_LinearRegression_nonlinear_verify(t *testing.T) {
+ variables := [][]float64{
+ {1, 2}, {2, 4}, {3, 8}, {4, 16}, {5, 32},
+ }
+
+ regression := newSimpleLinearRegression()
+ for _, v := range variables {
+ regression.AddPoint(v[0], v[1])
+ }
+ require.Less(t, regression.correlation, 0.9)
+}
+
+const float64EqualityThreshold = 1e-8
+
+func almostEqual(a, b float64) bool {
+ return math.Abs(a-b) <= float64EqualityThreshold
+}
diff --git a/pkg/fuzz/component/path.go b/pkg/fuzz/component/path.go
index 942c7e7e16..2671306de3 100644
--- a/pkg/fuzz/component/path.go
+++ b/pkg/fuzz/component/path.go
@@ -2,10 +2,18 @@ package component
import (
"context"
+ "strconv"
+ "strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/fuzz/dataformat"
"github.com/pkg/errors"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz/dataformat"
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
+ mapsutil "github.com/projectdiscovery/utils/maps"
+ urlutil "github.com/projectdiscovery/utils/url"
)
// Path is a component for a request Path
@@ -31,13 +39,18 @@ func (q *Path) Name() string {
// parsed component
func (q *Path) Parse(req *retryablehttp.Request) (bool, error) {
q.req = req
- q.value = NewValue(req.URL.Path)
+ q.value = NewValue("")
- parsed, err := dataformat.Get(dataformat.RawDataFormat).Decode(q.value.String())
- if err != nil {
- return false, err
+ splitted := strings.Split(req.URL.Path, "/")
+ values := make(map[string]interface{})
+ for i := range splitted {
+ pathTillNow := strings.Join(splitted[:i+1], "/")
+ if pathTillNow == "" {
+ continue
+ }
+ values[strconv.Itoa(i)] = pathTillNow
}
- q.value.SetParsed(parsed, dataformat.RawDataFormat)
+ q.value.SetParsed(dataformat.KVMap(values), "")
return true, nil
}
@@ -56,7 +69,8 @@ func (q *Path) Iterate(callback func(key string, value interface{}) error) (err
// SetValue sets a value in the component
// for a key
func (q *Path) SetValue(key string, value string) error {
- if !q.value.SetParsedValue(key, value) {
+ escaped := urlutil.ParamEncode(value)
+ if !q.value.SetParsedValue(key, escaped) {
return ErrSetValue
}
return nil
@@ -73,13 +87,45 @@ func (q *Path) Delete(key string) error {
// Rebuild returns a new request with the
// component rebuilt
func (q *Path) Rebuild() (*retryablehttp.Request, error) {
- encoded, err := q.value.Encode()
- if err != nil {
- return nil, errors.Wrap(err, "could not encode query")
+ originalValues := mapsutil.Map[string, any]{}
+ splitted := strings.Split(q.req.URL.Path, "/")
+ for i := range splitted {
+ pathTillNow := strings.Join(splitted[:i+1], "/")
+ if pathTillNow == "" {
+ continue
+ }
+ originalValues[strconv.Itoa(i)] = pathTillNow
}
+
+ originalPath := q.req.URL.Path
+ lengthSplitted := len(q.value.parsed.Map)
+ for i := lengthSplitted; i > 0; i-- {
+ key := strconv.Itoa(i)
+
+ original, ok := originalValues.GetOrDefault(key, "").(string)
+ if !ok {
+ continue
+ }
+
+ new, ok := q.value.parsed.Map.GetOrDefault(key, "").(string)
+ if !ok {
+ continue
+ }
+
+ if new == original {
+ // no need to replace
+ continue
+ }
+
+ originalPath = strings.Replace(originalPath, original, new, 1)
+ }
+
+ rebuiltPath := originalPath
+
+ // Clone the request and update the path
cloned := q.req.Clone(context.Background())
- if err := cloned.UpdateRelPath(encoded, true); err != nil {
- cloned.URL.RawPath = encoded
+ if err := cloned.UpdateRelPath(rebuiltPath, true); err != nil {
+ cloned.URL.RawPath = rebuiltPath
}
return cloned, nil
}
diff --git a/pkg/fuzz/component/path_test.go b/pkg/fuzz/component/path_test.go
index 859ffcde12..c47f81f4ff 100644
--- a/pkg/fuzz/component/path_test.go
+++ b/pkg/fuzz/component/path_test.go
@@ -28,10 +28,10 @@ func TestURLComponent(t *testing.T) {
return nil
})
- require.Equal(t, []string{"value"}, keys, "unexpected keys")
+ require.Equal(t, []string{"1"}, keys, "unexpected keys")
require.Equal(t, []string{"/testpath"}, values, "unexpected values")
- err = urlComponent.SetValue("value", "/newpath")
+ err = urlComponent.SetValue("1", "/newpath")
if err != nil {
t.Fatal(err)
}
@@ -40,7 +40,41 @@ func TestURLComponent(t *testing.T) {
if err != nil {
t.Fatal(err)
}
-
require.Equal(t, "/newpath", rebuilt.URL.Path, "unexpected URL path")
require.Equal(t, "https://example.com/newpath", rebuilt.URL.String(), "unexpected full URL")
}
+
+func TestURLComponent_NestedPaths(t *testing.T) {
+ path := NewPath()
+ req, err := retryablehttp.NewRequest(http.MethodGet, "https://example.com/user/753/profile", nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ found, err := path.Parse(req)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !found {
+ t.Fatal("expected path to be found")
+ }
+
+ isSet := false
+
+ _ = path.Iterate(func(key string, value interface{}) error {
+ if !isSet && value.(string) == "/user/753" {
+ isSet = true
+ if setErr := path.SetValue(key, "/user/753'"); setErr != nil {
+ t.Fatal(setErr)
+ }
+ }
+ return nil
+ })
+
+ newReq, err := path.Rebuild()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if newReq.URL.Path != "/user/753'/profile" {
+ t.Fatal("expected path to be modified")
+ }
+}
diff --git a/pkg/fuzz/component/query.go b/pkg/fuzz/component/query.go
index dc55e8fa80..22d6aabd12 100644
--- a/pkg/fuzz/component/query.go
+++ b/pkg/fuzz/component/query.go
@@ -61,6 +61,7 @@ func (q *Query) Iterate(callback func(key string, value interface{}) error) (err
// SetValue sets a value in the component
// for a key
func (q *Query) SetValue(key string, value string) error {
+ // Is this safe?
if !q.value.SetParsedValue(key, value) {
return ErrSetValue
}
diff --git a/pkg/fuzz/component/value.go b/pkg/fuzz/component/value.go
index be5bf0c566..614fd03002 100644
--- a/pkg/fuzz/component/value.go
+++ b/pkg/fuzz/component/value.go
@@ -73,12 +73,22 @@ func (v *Value) SetParsed(data dataformat.KV, dataFormat string) {
// SetParsedValue sets the parsed value for a key
// in the parsed map
-func (v *Value) SetParsedValue(key string, value string) bool {
+func (v *Value) SetParsedValue(key, value string) bool {
+ if key == "" {
+ return false
+ }
+
origValue := v.parsed.Get(key)
if origValue == nil {
v.parsed.Set(key, value)
return true
}
+
+ // TODO(dwisiswant0): I'm sure that this can be simplified because
+ // `dataformat.KV.*` is a type of `mapsutil.*` where the value is `any`. So,
+ // it looks like we won't type conversion here or even have its own methods
+ // inside `dataformat.KV`.
+
// If the value is a list, append to it
// otherwise replace it
switch v := origValue.(type) {
diff --git a/pkg/fuzz/dataformat/kv.go b/pkg/fuzz/dataformat/kv.go
index 72bd0da62e..3036a1c76c 100644
--- a/pkg/fuzz/dataformat/kv.go
+++ b/pkg/fuzz/dataformat/kv.go
@@ -13,7 +13,7 @@ import (
// if it's not important/significant (ex: json,xml) we use map
// this also allows us to iteratively implement ordered map
type KV struct {
- Map map[string]interface{}
+ Map mapsutil.Map[string, any]
OrderedMap *mapsutil.OrderedMap[string, any]
}
diff --git a/pkg/fuzz/dataformat/multipart.go b/pkg/fuzz/dataformat/multipart.go
index d7e40af10c..227025d22b 100644
--- a/pkg/fuzz/dataformat/multipart.go
+++ b/pkg/fuzz/dataformat/multipart.go
@@ -6,12 +6,19 @@ import (
"io"
"mime"
"mime/multipart"
+ "net/textproto"
mapsutil "github.com/projectdiscovery/utils/maps"
)
type MultiPartForm struct {
- boundary string
+ boundary string
+ filesMetadata map[string]FileMetadata
+}
+
+type FileMetadata struct {
+ ContentType string
+ Filename string
}
var (
@@ -41,11 +48,40 @@ func (m *MultiPartForm) Encode(data KV) (string, error) {
data.Iterate(func(key string, value any) bool {
var fw io.Writer
var err error
+
+ if filesArray, ok := value.([]interface{}); ok {
+ fileMetadata, ok := m.filesMetadata[key]
+ if !ok {
+ Itererr = fmt.Errorf("file metadata not found for key %s", key)
+ return false
+ }
+
+ for _, file := range filesArray {
+ h := make(textproto.MIMEHeader)
+ h.Set("Content-Disposition",
+ fmt.Sprintf(`form-data; name=%q; filename=%q`,
+ key, fileMetadata.Filename))
+ h.Set("Content-Type", fileMetadata.ContentType)
+
+ if fw, err = w.CreatePart(h); err != nil {
+ Itererr = err
+ return false
+ }
+
+ if _, err = fw.Write([]byte(file.(string))); err != nil {
+ Itererr = err
+ return false
+ }
+ }
+ return true
+ }
+
// Add field
if fw, err = w.CreateFormField(key); err != nil {
Itererr = err
return false
}
+
if _, err = fw.Write([]byte(value.(string))); err != nil {
Itererr = err
return false
@@ -98,6 +134,7 @@ func (m *MultiPartForm) Decode(data string) (KV, error) {
result.Set(key, values[0])
}
}
+ m.filesMetadata = make(map[string]FileMetadata)
for key, files := range form.File {
fileContents := []interface{}{}
for _, fileHeader := range files {
@@ -112,6 +149,11 @@ func (m *MultiPartForm) Decode(data string) (KV, error) {
return KV{}, err
}
fileContents = append(fileContents, buffer.String())
+
+ m.filesMetadata[key] = FileMetadata{
+ ContentType: fileHeader.Header.Get("Content-Type"),
+ Filename: fileHeader.Filename,
+ }
}
result.Set(key, fileContents)
}
diff --git a/pkg/fuzz/execute.go b/pkg/fuzz/execute.go
index 03ad26c4d0..46fb132a85 100644
--- a/pkg/fuzz/execute.go
+++ b/pkg/fuzz/execute.go
@@ -1,6 +1,7 @@
package fuzz
import (
+ "encoding/json"
"fmt"
"io"
"regexp"
@@ -15,6 +16,8 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/retryablehttp-go"
errorutil "github.com/projectdiscovery/utils/errors"
+ mapsutil "github.com/projectdiscovery/utils/maps"
+ sliceutil "github.com/projectdiscovery/utils/slice"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -45,6 +48,13 @@ type ExecuteRuleInput struct {
Values map[string]interface{}
// BaseRequest is the base http request for fuzzing rule
BaseRequest *retryablehttp.Request
+ // DisplayFuzzPoints is a flag to display fuzz points
+ DisplayFuzzPoints bool
+
+ // ApplyPayloadInitialTransformation is an optional function
+ // to transform the payload initially based on analyzer rules
+ ApplyPayloadInitialTransformation func(string, map[string]interface{}) string
+ AnalyzerParams map[string]interface{}
}
// GeneratedRequest is a single generated request for rule
@@ -59,6 +69,15 @@ type GeneratedRequest struct {
Component component.Component
// Parameter being fuzzed
Parameter string
+
+ // Key is the key for the request
+ Key string
+ // Value is the value for the request
+ Value string
+ // OriginalValue is the original value for the request
+ OriginalValue string
+ // OriginalPayload is the original payload for the request
+ OriginalPayload string
}
// Execute executes a fuzzing rule accepting a callback on which
@@ -76,8 +95,9 @@ func (rule *Rule) Execute(input *ExecuteRuleInput) (err error) {
var finalComponentList []component.Component
// match rule part with component name
+ displayDebugFuzzPoints := make(map[string]map[string]string)
for _, componentName := range component.Components {
- if rule.partType != requestPartType && rule.Part != componentName {
+ if !(rule.Part == componentName || sliceutil.Contains(rule.Parts, componentName) || rule.partType == requestPartType) {
continue
}
component := component.New(componentName)
@@ -89,12 +109,25 @@ func (rule *Rule) Execute(input *ExecuteRuleInput) (err error) {
if !discovered {
continue
}
+
// check rule applicable on this component
if !rule.checkRuleApplicableOnComponent(component) {
continue
}
+ // Debugging display for fuzz points
+ if input.DisplayFuzzPoints {
+ displayDebugFuzzPoints[componentName] = make(map[string]string)
+ _ = component.Iterate(func(key string, value interface{}) error {
+ displayDebugFuzzPoints[componentName][key] = fmt.Sprintf("%v", value)
+ return nil
+ })
+ }
finalComponentList = append(finalComponentList, component)
}
+ if len(displayDebugFuzzPoints) > 0 {
+ marshalled, _ := json.MarshalIndent(displayDebugFuzzPoints, "", " ")
+ gologger.Info().Msgf("[%s] Fuzz points for %s [%s]\n%s\n", rule.options.TemplateID, input.Input.MetaInput.Input, input.BaseRequest.Method, string(marshalled))
+ }
if len(finalComponentList) == 0 {
return ErrRuleNotApplicable.Msgf("no component matched on this rule")
@@ -147,14 +180,27 @@ mainLoop:
func (rule *Rule) evaluateVarsWithInteractsh(data map[string]interface{}, interactshUrls []string) (map[string]interface{}, []string) {
// Check if Interactsh options are configured
if rule.options.Interactsh != nil {
+ interactshUrlsMap := make(map[string]struct{})
+ for _, url := range interactshUrls {
+ interactshUrlsMap[url] = struct{}{}
+ }
+ interactshUrls = mapsutil.GetKeys(interactshUrlsMap)
// Iterate through the data to replace and evaluate variables with Interactsh URLs
for k, v := range data {
+ value := fmt.Sprint(v)
// Replace variables with Interactsh URLs and collect new URLs
- got, oastUrls := rule.options.Interactsh.Replace(fmt.Sprint(v), interactshUrls)
-
+ got, oastUrls := rule.options.Interactsh.Replace(value, interactshUrls)
+ if got != value {
+ data[k] = got
+ }
// Append new OAST URLs if any
if len(oastUrls) > 0 {
- interactshUrls = append(interactshUrls, oastUrls...)
+ for _, url := range oastUrls {
+ if _, ok := interactshUrlsMap[url]; !ok {
+ interactshUrlsMap[url] = struct{}{}
+ interactshUrls = append(interactshUrls, url)
+ }
+ }
}
// Evaluate the replaced data
evaluatedData, err := expressions.Evaluate(got, data)
@@ -184,7 +230,9 @@ func (rule *Rule) executeRuleValues(input *ExecuteRuleInput, ruleComponent compo
// if we are only fuzzing values
if len(rule.Fuzz.Value) > 0 {
for _, value := range rule.Fuzz.Value {
- if err := rule.executePartRule(input, ValueOrKeyValue{Value: value}, ruleComponent); err != nil {
+ originalPayload := value
+
+ if err := rule.executePartRule(input, ValueOrKeyValue{Value: value, OriginalPayload: originalPayload}, ruleComponent); err != nil {
if component.IsErrSetValue(err) {
// this are errors due to format restrictions
// ex: fuzzing string value in a json int field
@@ -225,7 +273,7 @@ func (rule *Rule) executeRuleValues(input *ExecuteRuleInput, ruleComponent compo
if err != nil {
return err
}
- if gotErr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent, ""); gotErr != nil {
+ if gotErr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent, "", "", "", "", "", ""); gotErr != nil {
return gotErr
}
}
@@ -261,8 +309,9 @@ func (rule *Rule) Compile(generator *generators.PayloadGenerator, options *proto
} else {
rule.partType = valueType
}
- } else {
- rule.partType = queryPartType
+ }
+ if rule.Part == "" && len(rule.Parts) == 0 {
+ return errors.Errorf("no part specified for rule")
}
if rule.Type != "" {
diff --git a/pkg/fuzz/frequency/tracker.go b/pkg/fuzz/frequency/tracker.go
new file mode 100644
index 0000000000..063912b4c0
--- /dev/null
+++ b/pkg/fuzz/frequency/tracker.go
@@ -0,0 +1,152 @@
+package frequency
+
+import (
+ "net"
+ "net/url"
+ "os"
+ "strings"
+ "sync"
+ "sync/atomic"
+
+ "github.com/bluele/gcache"
+ "github.com/projectdiscovery/gologger"
+)
+
+// Tracker implements a frequency tracker for a given input
+// which is used to determine uninteresting input parameters
+// which are not that interesting from fuzzing perspective for a template
+// and target combination.
+//
+// This is used to reduce the number of requests made during fuzzing
+// for parameters that are less likely to give results for a rule.
+type Tracker struct {
+ frequencies gcache.Cache
+ paramOccurenceThreshold int
+
+ isDebug bool
+}
+
+const (
+ DefaultMaxTrackCount = 10000
+ DefaultParamOccurenceThreshold = 10
+)
+
+type cacheItem struct {
+ errors atomic.Int32
+ sync.Once
+}
+
+// New creates a new frequency tracker with a given maximum
+// number of params to track in LRU fashion with a max error threshold
+func New(maxTrackCount, paramOccurenceThreshold int) *Tracker {
+ gc := gcache.New(maxTrackCount).ARC().Build()
+
+ var isDebug bool
+ if os.Getenv("FREQ_DEBUG") != "" {
+ isDebug = true
+ }
+ return &Tracker{
+ isDebug: isDebug,
+ frequencies: gc,
+ paramOccurenceThreshold: paramOccurenceThreshold,
+ }
+}
+
+func (t *Tracker) Close() {
+ t.frequencies.Purge()
+}
+
+// MarkParameter marks a parameter as frequently occuring once.
+//
+// The logic requires a parameter to be marked as frequently occuring
+// multiple times before it's considered as frequently occuring.
+func (t *Tracker) MarkParameter(parameter, target, template string) {
+ normalizedTarget := normalizeTarget(target)
+ key := getFrequencyKey(parameter, normalizedTarget, template)
+
+ if t.isDebug {
+ gologger.Verbose().Msgf("[%s] Marking %s as found uninteresting", template, key)
+ }
+
+ existingCacheItem, err := t.frequencies.GetIFPresent(key)
+ if err != nil || existingCacheItem == nil {
+ newItem := &cacheItem{errors: atomic.Int32{}}
+ newItem.errors.Store(1)
+ _ = t.frequencies.Set(key, newItem)
+ return
+ }
+ existingCacheItemValue := existingCacheItem.(*cacheItem)
+ existingCacheItemValue.errors.Add(1)
+
+ _ = t.frequencies.Set(key, existingCacheItemValue)
+}
+
+// IsParameterFrequent checks if a parameter is frequently occuring
+// in the input with no much results.
+func (t *Tracker) IsParameterFrequent(parameter, target, template string) bool {
+ normalizedTarget := normalizeTarget(target)
+ key := getFrequencyKey(parameter, normalizedTarget, template)
+
+ if t.isDebug {
+ gologger.Verbose().Msgf("[%s] Checking if %s is frequently found uninteresting", template, key)
+ }
+
+ existingCacheItem, err := t.frequencies.GetIFPresent(key)
+ if err != nil {
+ return false
+ }
+ existingCacheItemValue := existingCacheItem.(*cacheItem)
+
+ if existingCacheItemValue.errors.Load() >= int32(t.paramOccurenceThreshold) {
+ existingCacheItemValue.Do(func() {
+ gologger.Verbose().Msgf("[%s] Skipped %s from parameter for %s as found uninteresting %d times", template, parameter, target, existingCacheItemValue.errors.Load())
+ })
+ return true
+ }
+ return false
+}
+
+// UnmarkParameter unmarks a parameter as frequently occuring. This carries
+// more weight and resets the frequency counter for the parameter causing
+// it to be checked again. This is done when results are found.
+func (t *Tracker) UnmarkParameter(parameter, target, template string) {
+ normalizedTarget := normalizeTarget(target)
+ key := getFrequencyKey(parameter, normalizedTarget, template)
+
+ if t.isDebug {
+ gologger.Verbose().Msgf("[%s] Unmarking %s as frequently found uninteresting", template, key)
+ }
+
+ _ = t.frequencies.Remove(key)
+}
+
+func getFrequencyKey(parameter, target, template string) string {
+ var sb strings.Builder
+ sb.WriteString(target)
+ sb.WriteString(":")
+ sb.WriteString(template)
+ sb.WriteString(":")
+ sb.WriteString(parameter)
+ str := sb.String()
+ return str
+}
+
+func normalizeTarget(value string) string {
+ finalValue := value
+ if strings.HasPrefix(value, "http") {
+ if parsed, err := url.Parse(value); err == nil {
+ hostname := parsed.Host
+ finalPort := parsed.Port()
+ if finalPort == "" {
+ if parsed.Scheme == "https" {
+ finalPort = "443"
+ } else {
+ finalPort = "80"
+ }
+ hostname = net.JoinHostPort(parsed.Host, finalPort)
+ }
+ finalValue = hostname
+ }
+ }
+ return finalValue
+}
diff --git a/pkg/fuzz/fuzz.go b/pkg/fuzz/fuzz.go
index 036d09056d..0dcc820f66 100644
--- a/pkg/fuzz/fuzz.go
+++ b/pkg/fuzz/fuzz.go
@@ -4,8 +4,18 @@ import (
"regexp"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/fuzz/fuzz.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/fuzz/fuzz.go
+>>>>>>> projectdiscovery-main
)
// Rule is a single rule which describes how to fuzz the request
@@ -24,12 +34,27 @@ type Rule struct {
ruleType ruleType
// description: |
// Part is the part of request to fuzz.
- //
- // query fuzzes the query part of url. More parts will be added later.
// values:
// - "query"
+ // - "header"
+ // - "path"
+ // - "body"
+ // - "cookie"
+ // - "request"
Part string `yaml:"part,omitempty" json:"part,omitempty" jsonschema:"title=part of rule,description=Part of request rule to fuzz,enum=query,enum=header,enum=path,enum=body,enum=cookie,enum=request"`
partType partType
+ // description: |
+ // Parts is the list of parts to fuzz. If multiple parts need to be
+ // defined while excluding some, this should be used instead of singular part.
+ // values:
+ // - "query"
+ // - "header"
+ // - "path"
+ // - "body"
+ // - "cookie"
+ // - "request"
+ Parts []string `yaml:"parts,omitempty" json:"parts,omitempty" jsonschema:"title=parts of rule,description=Part of request rule to fuzz,enum=query,enum=header,enum=path,enum=body,enum=cookie,enum=request"`
+
// description: |
// Mode is the mode of fuzzing to perform.
//
diff --git a/pkg/fuzz/fuzz_test.go b/pkg/fuzz/fuzz_test.go
index dc0771259b..6ef2e39b08 100644
--- a/pkg/fuzz/fuzz_test.go
+++ b/pkg/fuzz/fuzz_test.go
@@ -7,7 +7,9 @@ import (
)
func TestRuleMatchKeyOrValue(t *testing.T) {
- rule := &Rule{}
+ rule := &Rule{
+ Part: "query",
+ }
err := rule.Compile(nil, nil)
require.NoError(t, err, "could not compile rule")
@@ -15,7 +17,7 @@ func TestRuleMatchKeyOrValue(t *testing.T) {
require.True(t, result, "could not get correct result")
t.Run("key", func(t *testing.T) {
- rule := &Rule{Keys: []string{"url"}}
+ rule := &Rule{Keys: []string{"url"}, Part: "query"}
err := rule.Compile(nil, nil)
require.NoError(t, err, "could not compile rule")
@@ -25,7 +27,7 @@ func TestRuleMatchKeyOrValue(t *testing.T) {
require.False(t, result, "could not get correct result")
})
t.Run("value", func(t *testing.T) {
- rule := &Rule{ValuesRegex: []string{`https?:\/\/?([-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b)*(\/[\/\d\w\.-]*)*(?:[\?])*(.+)*`}}
+ rule := &Rule{ValuesRegex: []string{`https?:\/\/?([-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b)*(\/[\/\d\w\.-]*)*(?:[\?])*(.+)*`}, Part: "query"}
err := rule.Compile(nil, nil)
require.NoError(t, err, "could not compile rule")
diff --git a/pkg/fuzz/parts.go b/pkg/fuzz/parts.go
index 4f6a19d561..8a47551118 100644
--- a/pkg/fuzz/parts.go
+++ b/pkg/fuzz/parts.go
@@ -2,6 +2,7 @@ package fuzz
import (
"io"
+ "strconv"
"strings"
"github.com/Explorer1092/nuclei/v3/pkg/fuzz/component"
@@ -9,6 +10,7 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/retryablehttp-go"
+ sliceutil "github.com/projectdiscovery/utils/slice"
)
// executePartRule executes part rules based on type
@@ -18,7 +20,7 @@ func (rule *Rule) executePartRule(input *ExecuteRuleInput, payload ValueOrKeyVal
// checkRuleApplicableOnComponent checks if a rule is applicable on given component
func (rule *Rule) checkRuleApplicableOnComponent(component component.Component) bool {
- if rule.Part != component.Name() {
+ if rule.Part != component.Name() && !sliceutil.Contains(rule.Parts, component.Name()) && rule.partType != requestPartType {
return false
}
foundAny := false
@@ -40,14 +42,14 @@ func (rule *Rule) executePartComponent(input *ExecuteRuleInput, payload ValueOrK
return rule.executePartComponentOnKV(input, payload, ruleComponent.Clone())
} else {
// for value only fuzzing
- return rule.executePartComponentOnValues(input, payload.Value, ruleComponent.Clone())
+ return rule.executePartComponentOnValues(input, payload.Value, payload.OriginalPayload, ruleComponent.Clone())
}
}
// executePartComponentOnValues executes this rule on a given component and payload
// this supports both single and multiple [ruleType] modes
// i.e if component has multiple values, they can be replaced once or all depending on mode
-func (rule *Rule) executePartComponentOnValues(input *ExecuteRuleInput, payloadStr string, ruleComponent component.Component) error {
+func (rule *Rule) executePartComponentOnValues(input *ExecuteRuleInput, payloadStr, originalPayload string, ruleComponent component.Component) error {
finalErr := ruleComponent.Iterate(func(key string, value interface{}) error {
valueStr := types.ToString(value)
if !rule.matchKeyOrValue(key, valueStr) {
@@ -55,8 +57,13 @@ func (rule *Rule) executePartComponentOnValues(input *ExecuteRuleInput, payloadS
return nil
}
- var evaluated string
+ var evaluated, originalEvaluated string
evaluated, input.InteractURLs = rule.executeEvaluate(input, key, valueStr, payloadStr, input.InteractURLs)
+ if input.ApplyPayloadInitialTransformation != nil {
+ evaluated = input.ApplyPayloadInitialTransformation(evaluated, input.AnalyzerParams)
+ originalEvaluated, _ = rule.executeEvaluate(input, key, valueStr, originalPayload, input.InteractURLs)
+ }
+
if err := ruleComponent.SetValue(key, evaluated); err != nil {
// gologger.Warning().Msgf("could not set value due to format restriction original(%s, %s[%T]) , new(%s,%s[%T])", key, valueStr, value, key, evaluated, evaluated)
return nil
@@ -68,7 +75,7 @@ func (rule *Rule) executePartComponentOnValues(input *ExecuteRuleInput, payloadS
return err
}
- if qerr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent, key); qerr != nil {
+ if qerr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent, key, valueStr, originalEvaluated, valueStr, key, evaluated); qerr != nil {
return qerr
}
// fmt.Printf("executed with value: %s\n", evaluated)
@@ -90,7 +97,7 @@ func (rule *Rule) executePartComponentOnValues(input *ExecuteRuleInput, payloadS
if err != nil {
return err
}
- if qerr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent, ""); qerr != nil {
+ if qerr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent, "", "", "", "", "", ""); qerr != nil {
err = qerr
return err
}
@@ -125,7 +132,7 @@ func (rule *Rule) executePartComponentOnKV(input *ExecuteRuleInput, payload Valu
return err
}
- if qerr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent, key); qerr != nil {
+ if qerr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent, key, value, "", "", "", ""); qerr != nil {
return err
}
@@ -144,13 +151,34 @@ func (rule *Rule) executePartComponentOnKV(input *ExecuteRuleInput, payload Valu
}
// execWithInput executes a rule with input via callback
-func (rule *Rule) execWithInput(input *ExecuteRuleInput, httpReq *retryablehttp.Request, interactURLs []string, component component.Component, parameter string) error {
+func (rule *Rule) execWithInput(input *ExecuteRuleInput, httpReq *retryablehttp.Request, interactURLs []string, component component.Component, parameter, parameterValue, originalPayload, originalValue, key, value string) error {
+ // If the parameter is a number, replace it with the parameter value
+ // or if the parameter is empty and the parameter value is not empty
+ // replace it with the parameter value
+ actualParameter := parameter
+ if _, err := strconv.Atoi(parameter); err == nil || (parameter == "" && parameterValue != "") {
+ actualParameter = parameterValue
+ }
+ // If the parameter is frequent, skip it if the option is enabled
+ if rule.options.FuzzParamsFrequency != nil {
+ if rule.options.FuzzParamsFrequency.IsParameterFrequent(
+ parameter,
+ httpReq.URL.String(),
+ rule.options.TemplateID,
+ ) {
+ return nil
+ }
+ }
request := GeneratedRequest{
- Request: httpReq,
- InteractURLs: interactURLs,
- DynamicValues: input.Values,
- Component: component,
- Parameter: parameter,
+ Request: httpReq,
+ InteractURLs: interactURLs,
+ DynamicValues: input.Values,
+ Component: component,
+ Parameter: actualParameter,
+ Key: key,
+ Value: value,
+ OriginalValue: originalValue,
+ OriginalPayload: originalPayload,
}
if !input.Callback(request) {
return types.ErrNoMoreRequests
@@ -163,9 +191,9 @@ func (rule *Rule) execWithInput(input *ExecuteRuleInput, httpReq *retryablehttp.
// for fuzzing.
func (rule *Rule) executeEvaluate(input *ExecuteRuleInput, _, value, payload string, interactshURLs []string) (string, []string) {
// TODO: Handle errors
- values := generators.MergeMaps(input.Values, map[string]interface{}{
+ values := generators.MergeMaps(rule.options.Variables.GetAll(), map[string]interface{}{
"value": value,
- }, rule.options.Options.Vars.AsMap(), rule.options.Variables.GetAll())
+ }, rule.options.Options.Vars.AsMap(), input.Values)
firstpass, _ := expressions.Evaluate(payload, values)
interactData, interactshURLs := rule.options.Interactsh.Replace(firstpass, interactshURLs)
evaluated, _ := expressions.Evaluate(interactData, values)
diff --git a/pkg/fuzz/type.go b/pkg/fuzz/type.go
index b8f6f3beef..04ba9c942d 100644
--- a/pkg/fuzz/type.go
+++ b/pkg/fuzz/type.go
@@ -19,6 +19,8 @@ var (
type ValueOrKeyValue struct {
Key string
Value string
+
+ OriginalPayload string
}
func (v *ValueOrKeyValue) IsKV() bool {
diff --git a/pkg/input/formats/openapi/examples.go b/pkg/input/formats/openapi/examples.go
index 35c4292cb1..9e7224ab73 100644
--- a/pkg/input/formats/openapi/examples.go
+++ b/pkg/input/formats/openapi/examples.go
@@ -18,7 +18,7 @@ func getSchemaExample(schema *openapi3.Schema) (interface{}, bool) {
return schema.Default, true
}
- if schema.Enum != nil && len(schema.Enum) > 0 {
+ if len(schema.Enum) > 0 {
return schema.Enum[0], true
}
return nil, false
@@ -175,9 +175,9 @@ func openAPIExample(schema *openapi3.Schema, cache map[*openapi3.Schema]*cachedS
}
switch {
- case schema.Type == "boolean":
+ case schema.Type.Is("boolean"):
return true, nil
- case schema.Type == "number", schema.Type == "integer":
+ case schema.Type.Is("number"), schema.Type.Is("integer"):
value := 0.0
if schema.Min != nil && *schema.Min > value {
@@ -208,11 +208,11 @@ func openAPIExample(schema *openapi3.Schema, cache map[*openapi3.Schema]*cachedS
value += float64(int(*schema.MultipleOf) - (int(value) % int(*schema.MultipleOf)))
}
- if schema.Type == "integer" {
+ if schema.Type.Is("integer") {
return int(value), nil
}
return value, nil
- case schema.Type == "string":
+ case schema.Type.Is("string"):
if ex := stringFormatExample(schema.Format); ex != "" {
return ex, nil
}
@@ -226,7 +226,7 @@ func openAPIExample(schema *openapi3.Schema, cache map[*openapi3.Schema]*cachedS
example = example[:*schema.MaxLength]
}
return example, nil
- case schema.Type == "array", schema.Items != nil:
+ case schema.Type.Is("array"), schema.Items != nil:
example := []interface{}{}
if schema.Items != nil && schema.Items.Value != nil {
@@ -242,7 +242,7 @@ func openAPIExample(schema *openapi3.Schema, cache map[*openapi3.Schema]*cachedS
}
}
return example, nil
- case schema.Type == "object", len(schema.Properties) > 0:
+ case schema.Type.Is("object"), len(schema.Properties) > 0:
example := map[string]interface{}{}
for k, v := range schema.Properties {
diff --git a/pkg/input/formats/openapi/generator.go b/pkg/input/formats/openapi/generator.go
index 6085680a0a..dba79a24f3 100644
--- a/pkg/input/formats/openapi/generator.go
+++ b/pkg/input/formats/openapi/generator.go
@@ -27,7 +27,8 @@ import (
)
const (
- globalAuth = "globalAuth"
+ globalAuth = "globalAuth"
+ DEFAULT_HTTP_SCHEME_HEADER = "Authorization"
)
// GenerateRequestsFromSchema generates http requests from an OpenAPI 3.0 document object
@@ -75,16 +76,26 @@ func GenerateRequestsFromSchema(schema *openapi3.T, opts formats.InputFormatOpti
for _, serverURL := range schema.Servers {
pathURL := serverURL.URL
+ // Split the server URL into baseURL and serverPath
+ u, err := url.Parse(pathURL)
+ if err != nil {
+ return errors.Wrap(err, "could not parse server url")
+ }
+ baseURL := fmt.Sprintf("%s://%s", u.Scheme, u.Host)
+ serverPath := u.Path
for path, v := range schema.Paths.Map() {
// a path item can have parameters
ops := v.Operations()
requestPath := path
+ if serverPath != "" {
+ requestPath = serverPath + path
+ }
for method, ov := range ops {
if err := generateRequestsFromOp(&generateReqOptions{
requiredOnly: opts.RequiredOnly,
method: method,
- pathURL: pathURL,
+ pathURL: baseURL,
requestPath: requestPath,
op: ov,
schema: schema,
@@ -193,7 +204,7 @@ func generateRequestsFromOp(opts *generateReqOptions) error {
paramValue = value.Schema.Value.Default
} else if value.Schema.Value.Example != nil {
paramValue = value.Schema.Value.Example
- } else if value.Schema.Value.Enum != nil && len(value.Schema.Value.Enum) > 0 {
+ } else if len(value.Schema.Value.Enum) > 0 {
paramValue = value.Schema.Value.Enum[0]
} else {
if !opts.opts.SkipFormatValidation {
@@ -406,7 +417,7 @@ schemaLabel:
return globalParams, nil
}
-// generateExampleFromSchema generates an example from a schema object
+// GenerateParameterFromSecurityScheme generates an example from a schema object
func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*openapi3.Parameter, error) {
if !generic.EqualsAny(scheme.Value.Type, "http", "apiKey") {
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
@@ -416,18 +427,20 @@ func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*o
if !generic.EqualsAny(scheme.Value.Scheme, "basic", "bearer") {
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme (%s) found in openapi file", scheme.Value.Scheme)
}
- if scheme.Value.Name == "" {
- return nil, errorutil.NewWithTag("openapi", "security scheme (%s) name is empty", scheme.Value.Scheme)
+ // HTTP authentication schemes basic or bearer use the Authorization header
+ headerName := scheme.Value.Name
+ if headerName == "" {
+ headerName = DEFAULT_HTTP_SCHEME_HEADER
}
// create parameters using the scheme
switch scheme.Value.Scheme {
case "basic":
- h := openapi3.NewHeaderParameter(scheme.Value.Name)
+ h := openapi3.NewHeaderParameter(headerName)
h.Required = true
h.Description = globalAuth // differentiator for normal variables and global auth
return h, nil
case "bearer":
- h := openapi3.NewHeaderParameter(scheme.Value.Name)
+ h := openapi3.NewHeaderParameter(headerName)
h.Required = true
h.Description = globalAuth // differentiator for normal variables and global auth
return h, nil
diff --git a/pkg/input/formats/swagger/swagger.go b/pkg/input/formats/swagger/swagger.go
index 61cc5e03aa..7bc191d345 100644
--- a/pkg/input/formats/swagger/swagger.go
+++ b/pkg/input/formats/swagger/swagger.go
@@ -2,6 +2,7 @@ package swagger
import (
"encoding/json"
+ "io"
"os"
"path"
@@ -10,7 +11,13 @@ import (
"github.com/getkin/kin-openapi/openapi2"
"github.com/getkin/kin-openapi/openapi3"
"github.com/pkg/errors"
+<<<<<<< HEAD
"gopkg.in/yaml.v2"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
+ "github.com/projectdiscovery/nuclei/v3/pkg/input/formats/openapi"
+ "github.com/invopop/yaml"
+>>>>>>> projectdiscovery-main
"github.com/getkin/kin-openapi/openapi2conv"
)
@@ -49,7 +56,11 @@ func (j *SwaggerFormat) Parse(input string, resultsCb formats.ParseReqRespCallba
ext := path.Ext(input)
if ext == ".yaml" || ext == ".yml" {
- err = yaml.NewDecoder(file).Decode(schemav2)
+ data, err_data := io.ReadAll(file)
+ if err_data != nil {
+ return errors.Wrap(err, "could not read data file")
+ }
+ err = yaml.Unmarshal(data, schemav2)
} else {
err = json.NewDecoder(file).Decode(schemav2)
}
diff --git a/pkg/input/formats/swagger/swagger_test.go b/pkg/input/formats/swagger/swagger_test.go
index fbd1c73ad5..995fc5eb03 100644
--- a/pkg/input/formats/swagger/swagger_test.go
+++ b/pkg/input/formats/swagger/swagger_test.go
@@ -27,8 +27,8 @@ func TestSwaggerAPIParser(t *testing.T) {
}
expectedURLs := []string{
- "https://localhost/users",
- "https://localhost/users/1?test=asc",
+ "https://localhost/v1/users",
+ "https://localhost/v1/users/1?test=asc",
}
require.ElementsMatch(t, gotMethodsToURLs, expectedURLs, "could not get swagger urls")
}
diff --git a/pkg/input/provider/http/multiformat.go b/pkg/input/provider/http/multiformat.go
index 2152f47350..a4984328e2 100644
--- a/pkg/input/provider/http/multiformat.go
+++ b/pkg/input/provider/http/multiformat.go
@@ -66,9 +66,10 @@ func (i *HttpInputProvider) Count() int64 {
// Iterate over all inputs in order
func (i *HttpInputProvider) Iterate(callback func(value *contextargs.MetaInput) bool) {
err := i.format.Parse(i.inputFile, func(request *types.RequestResponse) bool {
- return callback(&contextargs.MetaInput{
- ReqResp: request,
- })
+ metaInput := contextargs.NewMetaInput()
+ metaInput.ReqResp = request
+ metaInput.Input = request.URL.String()
+ return callback(metaInput)
})
if err != nil {
gologger.Warning().Msgf("Could not parse input file while iterating: %s\n", err)
diff --git a/pkg/input/provider/interface.go b/pkg/input/provider/interface.go
index b46d7928a1..aeaebc1fc7 100644
--- a/pkg/input/provider/interface.go
+++ b/pkg/input/provider/interface.go
@@ -14,6 +14,7 @@ import (
configTypes "github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/gologger"
errorutil "github.com/projectdiscovery/utils/errors"
+ stringsutil "github.com/projectdiscovery/utils/strings"
)
var (
@@ -32,7 +33,7 @@ func IsErrNotImplemented(err error) bool {
if err == nil {
return false
}
- if strings.Contains(err.Error(), "provider") && strings.Contains(err.Error(), "does not implement") {
+ if stringsutil.ContainsAll(err.Error(), "provider", "does not implement") {
return true
}
return false
@@ -120,7 +121,7 @@ func NewInputProvider(opts InputOptions) (InputProvider, error) {
}
}
-// SupportedFormats returns all supported input formats of nuclei
+// SupportedInputFormats returns all supported input formats of nuclei
func SupportedInputFormats() string {
return "list, " + http.SupportedFormats()
}
diff --git a/pkg/input/provider/list/hmap.go b/pkg/input/provider/list/hmap.go
index 0d5b5d2c24..8f62a974fd 100644
--- a/pkg/input/provider/list/hmap.go
+++ b/pkg/input/provider/list/hmap.go
@@ -26,6 +26,28 @@ import (
"github.com/projectdiscovery/hmap/filekv"
"github.com/projectdiscovery/hmap/store/hybrid"
"github.com/projectdiscovery/mapcidr/asn"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/core/inputs/hybrid/hmap.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolstate"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/uncover"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/uncover"
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+=======
+ providerTypes "github.com/projectdiscovery/nuclei/v3/pkg/input/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/uncover"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/expand"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/input/provider/list/hmap.go
+>>>>>>> projectdiscovery-main
uncoverlib "github.com/projectdiscovery/uncover"
fileutil "github.com/projectdiscovery/utils/file"
iputil "github.com/projectdiscovery/utils/ip"
@@ -123,7 +145,7 @@ func (i *ListInputProvider) Iterate(callback func(value *contextargs.MetaInput)
})
}
callbackFunc := func(k, _ []byte) error {
- metaInput := &contextargs.MetaInput{}
+ metaInput := contextargs.NewMetaInput()
if err := metaInput.Unmarshal(string(k)); err != nil {
return err
}
@@ -162,14 +184,16 @@ func (i *ListInputProvider) Set(value string) {
}
return fmt.Sprintf("got empty hostname for %v skipping ip selection", URL)
})
- metaInput := &contextargs.MetaInput{Input: URL}
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = URL
i.setItem(metaInput)
return
}
// Check if input is ip or hostname
if iputil.IsIP(urlx.Hostname()) {
- metaInput := &contextargs.MetaInput{Input: URL}
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = URL
i.setItem(metaInput)
return
}
@@ -190,7 +214,9 @@ func (i *ListInputProvider) Set(value string) {
if ip == "" {
continue
}
- metaInput := &contextargs.MetaInput{Input: value, CustomIP: ip}
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = value
+ metaInput.CustomIP = ip
i.setItem(metaInput)
}
return
@@ -220,11 +246,13 @@ func (i *ListInputProvider) Set(value string) {
}
for _, ip := range ips {
+ metaInput := contextargs.NewMetaInput()
if ip != "" {
- metaInput := &contextargs.MetaInput{Input: URL, CustomIP: ip}
+ metaInput.Input = URL
+ metaInput.CustomIP = ip
i.setItem(metaInput)
} else {
- metaInput := &contextargs.MetaInput{Input: URL}
+ metaInput.Input = URL
i.setItem(metaInput)
}
}
@@ -362,7 +390,8 @@ func (i *ListInputProvider) scanInputFromReader(reader io.Reader) {
// isExcluded checks if a URL is in the exclusion list
func (i *ListInputProvider) isExcluded(URL string) bool {
- metaInput := &contextargs.MetaInput{Input: URL}
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = URL
key, err := metaInput.MarshalString()
if err != nil {
gologger.Warning().Msgf("%s\n", err)
@@ -395,14 +424,16 @@ func (i *ListInputProvider) Del(value string) {
}
return fmt.Sprintf("got empty hostname for %v skipping ip selection", URL)
})
- metaInput := &contextargs.MetaInput{Input: URL}
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = URL
i.delItem(metaInput)
return
}
// Check if input is ip or hostname
if iputil.IsIP(urlx.Hostname()) {
- metaInput := &contextargs.MetaInput{Input: URL}
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = URL
i.delItem(metaInput)
return
}
@@ -423,7 +454,9 @@ func (i *ListInputProvider) Del(value string) {
if ip == "" {
continue
}
- metaInput := &contextargs.MetaInput{Input: value, CustomIP: ip}
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = value
+ metaInput.CustomIP = ip
i.delItem(metaInput)
}
return
@@ -453,11 +486,13 @@ func (i *ListInputProvider) Del(value string) {
}
for _, ip := range ips {
+ metaInput := contextargs.NewMetaInput()
if ip != "" {
- metaInput := &contextargs.MetaInput{Input: URL, CustomIP: ip}
+ metaInput.Input = URL
+ metaInput.CustomIP = ip
i.delItem(metaInput)
} else {
- metaInput := &contextargs.MetaInput{Input: URL}
+ metaInput.Input = URL
i.delItem(metaInput)
}
}
@@ -532,7 +567,8 @@ func (i *ListInputProvider) addTargets(targets []string) {
func (i *ListInputProvider) removeTargets(targets []string) {
for _, target := range targets {
- metaInput := &contextargs.MetaInput{Input: target}
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = target
i.delItem(metaInput)
}
}
diff --git a/pkg/input/provider/list/hmap_test.go b/pkg/input/provider/list/hmap_test.go
index 33f462a5ae..7c19d4e3bc 100644
--- a/pkg/input/provider/list/hmap_test.go
+++ b/pkg/input/provider/list/hmap_test.go
@@ -13,7 +13,21 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/utils/expand"
"github.com/miekg/dns"
"github.com/projectdiscovery/hmap/store/hybrid"
+<<<<<<< HEAD
"github.com/projectdiscovery/utils/auth/pdcp"
+=======
+<<<<<<< HEAD:v2/pkg/core/inputs/hybrid/hmap_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolstate"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/expand"
+ "github.com/projectdiscovery/utils/auth/pdcp"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/input/provider/list/hmap_test.go
+>>>>>>> projectdiscovery-main
"github.com/stretchr/testify/require"
)
@@ -40,7 +54,7 @@ func Test_expandCIDR(t *testing.T) {
// scan
got := []string{}
input.hostMap.Scan(func(k, _ []byte) error {
- var metainput contextargs.MetaInput
+ metainput := contextargs.NewMetaInput()
if err := metainput.Unmarshal(string(k)); err != nil {
return err
}
@@ -141,7 +155,7 @@ func Test_scanallips_normalizeStoreInputValue(t *testing.T) {
// scan
got := []string{}
input.hostMap.Scan(func(k, v []byte) error {
- var metainput contextargs.MetaInput
+ metainput := contextargs.NewMetaInput()
if err := metainput.Unmarshal(string(k)); err != nil {
return err
}
@@ -184,7 +198,7 @@ func Test_expandASNInputValue(t *testing.T) {
// scan the hmap
got := []string{}
input.hostMap.Scan(func(k, v []byte) error {
- var metainput contextargs.MetaInput
+ metainput := contextargs.NewMetaInput()
if err := metainput.Unmarshal(string(k)); err != nil {
return err
}
diff --git a/pkg/input/provider/simple.go b/pkg/input/provider/simple.go
index 58c5ff2d29..491a09e6ce 100644
--- a/pkg/input/provider/simple.go
+++ b/pkg/input/provider/simple.go
@@ -43,7 +43,9 @@ func (s *SimpleInputProvider) Iterate(callback func(value *contextargs.MetaInput
// Set adds an item to the input provider
func (s *SimpleInputProvider) Set(value string) {
- s.Inputs = append(s.Inputs, &contextargs.MetaInput{Input: value})
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = value
+ s.Inputs = append(s.Inputs, metaInput)
}
// SetWithProbe adds an item to the input provider with HTTP probing
@@ -52,13 +54,17 @@ func (s *SimpleInputProvider) SetWithProbe(value string, probe types.InputLivene
if err != nil {
return err
}
- s.Inputs = append(s.Inputs, &contextargs.MetaInput{Input: probedValue})
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = probedValue
+ s.Inputs = append(s.Inputs, metaInput)
return nil
}
// SetWithExclusions adds an item to the input provider if it doesn't match any of the exclusions
func (s *SimpleInputProvider) SetWithExclusions(value string) error {
- s.Inputs = append(s.Inputs, &contextargs.MetaInput{Input: value})
+ metaInput := contextargs.NewMetaInput()
+ metaInput.Input = value
+ s.Inputs = append(s.Inputs, metaInput)
return nil
}
diff --git a/pkg/input/transform.go b/pkg/input/transform.go
index a61b21de23..8144a2f874 100644
--- a/pkg/input/transform.go
+++ b/pkg/input/transform.go
@@ -7,6 +7,14 @@ import (
templateTypes "github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/hmap/store/hybrid"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/input/input.go
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+=======
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/input/transform.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
"github.com/projectdiscovery/utils/ports"
stringsutil "github.com/projectdiscovery/utils/strings"
@@ -47,6 +55,8 @@ func (h *Helper) Transform(input string, protocol templateTypes.ProtocolType) st
return h.convertInputToType(input, typeHostWithOptionalPort, "")
case templateTypes.WebsocketProtocol:
return h.convertInputToType(input, typeWebsocket, "")
+ case templateTypes.SSLProtocol:
+ return h.convertInputToType(input, typeHostWithPort, "443")
}
return input
}
@@ -94,6 +104,8 @@ func (h *Helper) convertInputToType(input string, inputType inputType, defaultPo
if _, err := filepath.Match(input, ""); err != filepath.ErrBadPattern && !isURL {
return input
}
+ // if none of these satisfy the condition return empty
+ return ""
case typeHostOnly:
if hasHost {
return host
@@ -111,6 +123,10 @@ func (h *Helper) convertInputToType(input string, inputType inputType, defaultPo
return string(probed)
}
}
+ // try to parse it as absolute url and return
+ if absUrl, err := urlutil.ParseAbsoluteURL(input, false); err == nil {
+ return absUrl.String()
+ }
case typeHostWithPort, typeHostWithOptionalPort:
if hasHost && hasPort {
return net.JoinHostPort(host, port)
@@ -128,6 +144,9 @@ func (h *Helper) convertInputToType(input string, inputType inputType, defaultPo
if uri != nil && stringsutil.EqualFoldAny(uri.Scheme, "ws", "wss") {
return input
}
+ // empty if prefix is not given
+ return ""
}
- return ""
+ // do not return empty
+ return input
}
diff --git a/pkg/input/transform_test.go b/pkg/input/transform_test.go
index 76cabd3b53..699d877729 100644
--- a/pkg/input/transform_test.go
+++ b/pkg/input/transform_test.go
@@ -30,7 +30,7 @@ func TestConvertInputToType(t *testing.T) {
{"https://google.com:443", typeHostOnly, "google.com", ""},
// url
- {"test.com", typeURL, "", ""},
+ {"test.com", typeURL, "test.com", ""},
{"google.com", typeURL, "https://google.com", ""},
{"https://google.com", typeURL, "https://google.com", ""},
@@ -43,7 +43,7 @@ func TestConvertInputToType(t *testing.T) {
{"input_test.*", typeFilepath, "input_test.*", ""},
// host-port
- {"google.com", typeHostWithPort, "", ""},
+ {"google.com", typeHostWithPort, "google.com", ""},
{"google.com:443", typeHostWithPort, "google.com:443", ""},
{"https://google.com", typeHostWithPort, "google.com:443", ""},
{"https://google.com:443", typeHostWithPort, "google.com:443", ""},
diff --git a/pkg/input/types/http.go b/pkg/input/types/http.go
index dbc5c9e325..0620d85f8c 100644
--- a/pkg/input/types/http.go
+++ b/pkg/input/types/http.go
@@ -123,11 +123,15 @@ func (rr *RequestResponse) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &m); err != nil {
return err
}
- urlStr, ok := m["url"]
+ urlStrRaw, ok := m["url"]
if !ok {
return fmt.Errorf("missing url in request response")
}
- parsed, err := urlutil.ParseAbsoluteURL(string(urlStr), false)
+ var urlStr string
+ if err := json.Unmarshal(urlStrRaw, &urlStr); err != nil {
+ return err
+ }
+ parsed, err := urlutil.ParseAbsoluteURL(urlStr, false)
if err != nil {
return err
}
diff --git a/pkg/input/types/http_test.go b/pkg/input/types/http_test.go
index d674c56353..9525c516e0 100644
--- a/pkg/input/types/http_test.go
+++ b/pkg/input/types/http_test.go
@@ -65,3 +65,26 @@ func TestParseHttpRequest(t *testing.T) {
})
}
}
+
+func TestUnmarshalJSON(t *testing.T) {
+ tests := []struct {
+ name string
+ rawJSONStr string
+ expectedURLStr string
+ }{
+ {"basic url", `{"url": "example.com"}`, "example.com"},
+ {"basic url with scheme", `{"url": "http://example.com"}`, "http://example.com"},
+ }
+ for _, tc := range tests {
+ t.Run(tc.name, func(t *testing.T) {
+ var rr RequestResponse
+ err := rr.UnmarshalJSON([]byte(tc.rawJSONStr))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if tc.expectedURLStr != "" {
+ require.Equal(t, rr.URL.String(), tc.expectedURLStr)
+ }
+ })
+ }
+}
diff --git a/pkg/installer/template.go b/pkg/installer/template.go
index 742d9643d0..11fcc654e6 100644
--- a/pkg/installer/template.go
+++ b/pkg/installer/template.go
@@ -17,6 +17,16 @@ import (
"github.com/charmbracelet/glamour"
"github.com/olekukonko/tablewriter"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/internal/installer/template.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/external/customtemplates"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/external/customtemplates"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/installer/template.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
stringsutil "github.com/projectdiscovery/utils/strings"
diff --git a/pkg/installer/template_test.go b/pkg/installer/template_test.go
index 75abe8e148..cf385aaa09 100644
--- a/pkg/installer/template_test.go
+++ b/pkg/installer/template_test.go
@@ -6,7 +6,15 @@ import (
"strings"
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
+=======
+<<<<<<< HEAD:v2/internal/installer/template_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/installer/template_test.go
+>>>>>>> projectdiscovery-main
"github.com/stretchr/testify/require"
)
diff --git a/pkg/installer/util.go b/pkg/installer/util.go
index 81dde4472f..02c48932e4 100644
--- a/pkg/installer/util.go
+++ b/pkg/installer/util.go
@@ -14,6 +14,14 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Masterminds/semver/v3"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/internal/installer/util.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/installer/util.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
)
diff --git a/pkg/installer/versioncheck.go b/pkg/installer/versioncheck.go
index c1fa000f6e..23da061cec 100644
--- a/pkg/installer/versioncheck.go
+++ b/pkg/installer/versioncheck.go
@@ -7,8 +7,17 @@ import (
"os"
"sync"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/gologger"
+=======
+<<<<<<< HEAD:v2/internal/installer/versioncheck.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+=======
+ "github.com/projectdiscovery/gologger"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/installer/versioncheck.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
updateutils "github.com/projectdiscovery/utils/update"
)
@@ -87,6 +96,7 @@ func doVersionCheck(isSDK bool) error {
// and according our config we have idle connections which are shown as leaked by goleak in tests
// i.e we close all idle connections after our use and it doesn't affect any other part of the code
defer retryableHttpClient.HTTPClient.CloseIdleConnections()
+
resp, err := retryableHttpClient.Get(pdtmNucleiVersionEndpoint + "?" + getpdtmParams(isSDK))
if err != nil {
return err
diff --git a/pkg/installer/versioncheck_test.go b/pkg/installer/versioncheck_test.go
index edf4e136ae..9e74ef9a5f 100644
--- a/pkg/installer/versioncheck_test.go
+++ b/pkg/installer/versioncheck_test.go
@@ -3,7 +3,15 @@ package installer
import (
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
+=======
+<<<<<<< HEAD:v2/internal/installer/versioncheck_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/installer/versioncheck_test.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/utils/generic"
"github.com/stretchr/testify/require"
)
diff --git a/pkg/js/compiler/compiler.go b/pkg/js/compiler/compiler.go
index aaa4b7afd4..0c476e59b8 100644
--- a/pkg/js/compiler/compiler.go
+++ b/pkg/js/compiler/compiler.go
@@ -4,15 +4,26 @@ package compiler
import (
"context"
"fmt"
- "time"
"github.com/dop251/goja"
+ "github.com/kitabisa/go-ci"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> projectdiscovery-main
contextutil "github.com/projectdiscovery/utils/context"
+ "github.com/projectdiscovery/utils/errkit"
stringsutil "github.com/projectdiscovery/utils/strings"
)
+var (
+ // ErrJSExecDeadline is the error returned when alloted time for script execution exceeds
+ ErrJSExecDeadline = errkit.New("js engine execution deadline exceeded").SetKind(errkit.ErrKindDeadline).Build()
+)
+
// Compiler provides a runtime to execute goja runtime
// based javascript scripts efficiently while also
// providing them access to custom modules defined in libs/.
@@ -32,13 +43,13 @@ type ExecuteOptions struct {
// Cleanup is extra cleanup function to be called after execution
Cleanup func(runtime *goja.Runtime)
- /// Timeout for this script execution
- Timeout int
// Source is original source of the script
Source *string
Context context.Context
+ TimeoutVariants *types.Timeouts
+
// Manually exported objects
exports map[string]interface{}
}
@@ -49,6 +60,11 @@ type ExecuteArgs struct {
TemplateCtx map[string]interface{} // templateCtx contains template scoped variables
}
+// Map returns a merged map of the TemplateCtx and Args fields.
+func (e *ExecuteArgs) Map() map[string]interface{} {
+ return generators.MergeMaps(e.TemplateCtx, e.Args)
+}
+
// NewExecuteArgs returns a new execute arguments.
func NewExecuteArgs() *ExecuteArgs {
return &ExecuteArgs{
@@ -60,12 +76,24 @@ func NewExecuteArgs() *ExecuteArgs {
// ExecuteResult is the result of executing a script.
type ExecuteResult map[string]interface{}
+// Map returns the map representation of the ExecuteResult
+func (e ExecuteResult) Map() map[string]interface{} {
+ if e == nil {
+ return make(map[string]interface{})
+ }
+ return e
+}
+
+// NewExecuteResult returns a new execute result instance
func NewExecuteResult() ExecuteResult {
return make(map[string]interface{})
}
// GetSuccess returns whether the script was successful or not.
func (e ExecuteResult) GetSuccess() bool {
+ if e == nil {
+ return false
+ }
val, ok := e["success"].(bool)
if !ok {
return false
@@ -73,15 +101,6 @@ func (e ExecuteResult) GetSuccess() bool {
return val
}
-// Execute executes a script with the default options.
-func (c *Compiler) Execute(code string, args *ExecuteArgs) (ExecuteResult, error) {
- p, err := WrapScriptNCompile(code, false)
- if err != nil {
- return nil, err
- }
- return c.ExecuteWithOptions(p, args, &ExecuteOptions{Context: context.Background()})
-}
-
// ExecuteWithOptions executes a script with the provided options.
func (c *Compiler) ExecuteWithOptions(program *goja.Program, args *ExecuteArgs, opts *ExecuteOptions) (ExecuteResult, error) {
if opts == nil {
@@ -100,26 +119,34 @@ func (c *Compiler) ExecuteWithOptions(program *goja.Program, args *ExecuteArgs,
// merge all args into templatectx
args.TemplateCtx = generators.MergeMaps(args.TemplateCtx, args.Args)
- if opts.Timeout <= 0 || opts.Timeout > 180 {
- // some js scripts can take longer time so allow configuring timeout
- // from template but keep it within sane limits (180s)
- opts.Timeout = JsProtocolTimeout
- }
-
// execute with context and timeout
- ctx, cancel := context.WithTimeout(opts.Context, time.Duration(opts.Timeout)*time.Second)
+
+ ctx, cancel := context.WithTimeoutCause(opts.Context, opts.TimeoutVariants.JsCompilerExecutionTimeout, ErrJSExecDeadline)
defer cancel()
// execute the script
results, err := contextutil.ExecFuncWithTwoReturns(ctx, func() (val goja.Value, err error) {
+ // TODO(dwisiswant0): remove this once we get the RCA.
defer func() {
+ if ci.IsCI() {
+ return
+ }
+
if r := recover(); r != nil {
err = fmt.Errorf("panic: %v", r)
}
}()
+
return ExecuteProgram(program, args, opts)
})
if err != nil {
- return nil, err
+ if val, ok := err.(*goja.Exception); ok {
+ if x := val.Unwrap(); x != nil {
+ err = x
+ }
+ }
+ e := NewExecuteResult()
+ e["error"] = err.Error()
+ return e, err
}
var res ExecuteResult
if opts.exports != nil {
diff --git a/pkg/js/compiler/compiler_test.go b/pkg/js/compiler/compiler_test.go
index ca09e7782e..3a43a4016e 100644
--- a/pkg/js/compiler/compiler_test.go
+++ b/pkg/js/compiler/compiler_test.go
@@ -1,11 +1,14 @@
package compiler
import (
+ "context"
"strings"
"testing"
+ "time"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/gologger/levels"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
)
func TestNewCompilerConsoleDebug(t *testing.T) {
@@ -18,23 +21,19 @@ func TestNewCompilerConsoleDebug(t *testing.T) {
})
compiler := New()
- _, err := compiler.Execute("console.log('hello world');", NewExecuteArgs())
+ p, err := WrapScriptNCompile("console.log('hello world');", false)
if err != nil {
t.Fatal(err)
}
- if !strings.HasSuffix(gotString, "hello world") {
- t.Fatalf("console.log not working, got=%v", gotString)
- }
-}
-func TestExecuteResultGetSuccess(t *testing.T) {
- compiler := New()
- result, err := compiler.Execute("1+1 == 2", NewExecuteArgs())
+ _, err = compiler.ExecuteWithOptions(p, NewExecuteArgs(), &ExecuteOptions{Context: context.Background(),
+ TimeoutVariants: &types.Timeouts{JsCompilerExecutionTimeout: time.Duration(20) * time.Second}},
+ )
if err != nil {
t.Fatal(err)
}
- if result.GetSuccess() != true {
- t.Fatalf("expected true, got=%v", result.GetSuccess())
+ if !strings.HasSuffix(gotString, "hello world") {
+ t.Fatalf("console.log not working, got=%v", gotString)
}
}
diff --git a/pkg/js/compiler/init.go b/pkg/js/compiler/init.go
index 5ce9d0bff1..dc83a14bdb 100644
--- a/pkg/js/compiler/init.go
+++ b/pkg/js/compiler/init.go
@@ -1,27 +1,27 @@
package compiler
+<<<<<<< HEAD
import "github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+import (
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+)
+>>>>>>> projectdiscovery-main
// jsprotocolInit
var (
- // Per Execution Javascript timeout in seconds
- JsProtocolTimeout = 10
PoolingJsVmConcurrency = 100
NonPoolingVMConcurrency = 20
)
// Init initializes the javascript protocol
func Init(opts *types.Options) error {
- if opts.Timeout < 10 {
- // keep existing 10s timeout
- return nil
- }
+
if opts.JsConcurrency < 100 {
// 100 is reasonable default
opts.JsConcurrency = 100
}
- JsProtocolTimeout = opts.Timeout
PoolingJsVmConcurrency = opts.JsConcurrency
PoolingJsVmConcurrency -= NonPoolingVMConcurrency
return nil
diff --git a/pkg/js/compiler/pool.go b/pkg/js/compiler/pool.go
index 8c88b2cc71..43821a9fe4 100644
--- a/pkg/js/compiler/pool.go
+++ b/pkg/js/compiler/pool.go
@@ -35,6 +35,7 @@ import (
"github.com/dop251/goja"
"github.com/dop251/goja_nodejs/console"
"github.com/dop251/goja_nodejs/require"
+ "github.com/kitabisa/go-ci"
"github.com/projectdiscovery/gologger"
stringsutil "github.com/projectdiscovery/utils/strings"
syncutil "github.com/projectdiscovery/utils/sync"
@@ -84,11 +85,18 @@ func executeWithRuntime(runtime *goja.Runtime, p *goja.Program, args *ExecuteArg
opts.Cleanup(runtime)
}
}()
+
+ // TODO(dwisiswant0): remove this once we get the RCA.
defer func() {
+ if ci.IsCI() {
+ return
+ }
+
if r := recover(); r != nil {
err = fmt.Errorf("panic: %s", r)
}
}()
+
// set template ctx
_ = runtime.Set("template", args.TemplateCtx)
// set args
@@ -241,5 +249,5 @@ func stringify(gojaValue goja.Value, runtime *goja.Runtime) string {
}
}
// for everything else stringify
- return fmt.Sprintf("%v", value)
+ return fmt.Sprintf("%+v", value)
}
diff --git a/pkg/js/devtools/bindgen/generator.go b/pkg/js/devtools/bindgen/generator.go
index 192eb0253d..ea6b44f924 100644
--- a/pkg/js/devtools/bindgen/generator.go
+++ b/pkg/js/devtools/bindgen/generator.go
@@ -249,7 +249,7 @@ func identifyGenDecl(pkg *ast.Package, decl *ast.GenDecl, data *TemplateData) {
if !spec.Names[0].IsExported() {
continue
}
- if spec.Values == nil || len(spec.Values) == 0 {
+ if len(spec.Values) == 0 {
continue
}
data.PackageVars[spec.Names[0].Name] = spec.Names[0].Name
diff --git a/pkg/js/devtools/scrapefuncs/README.md b/pkg/js/devtools/scrapefuncs/README.md
index de7c29ce2c..ecde2d5ab4 100644
--- a/pkg/js/devtools/scrapefuncs/README.md
+++ b/pkg/js/devtools/scrapefuncs/README.md
@@ -42,7 +42,11 @@ Description: getNetworkPort registers defaultPort and returns defaultPort if it
Name: isPortOpen
Signatures: "isPortOpen(host string, port string, [timeout int]) bool"
-Description: isPortOpen checks if given port is open on host. timeout is optional and defaults to 5 seconds
+Description: isPortOpen checks if given TCP port is open on host. timeout is optional and defaults to 5 seconds
+
+Name: isUDPPortOpen
+Signatures: "isUDPPortOpen(host string, port string, [timeout int]) bool"
+Description: isUDPPortOpen checks if the given UDP port is open on the host. Timeout is optional and defaults to 5 seconds.
Name: ToBytes
Signatures: "ToBytes(...interface{}) []byte"
diff --git a/pkg/js/devtools/tsgen/parser.go b/pkg/js/devtools/tsgen/parser.go
index e285b5f007..9f7d81ed7b 100644
--- a/pkg/js/devtools/tsgen/parser.go
+++ b/pkg/js/devtools/tsgen/parser.go
@@ -489,7 +489,7 @@ func (p *EntityParser) extractVarsNConstants() {
if !spec.Names[0].IsExported() {
continue
}
- if spec.Values == nil || len(spec.Values) == 0 {
+ if len(spec.Values) == 0 {
continue
}
// get comments or description
diff --git a/pkg/js/generated/go/libldap/ldap.go b/pkg/js/generated/go/libldap/ldap.go
index 6692e71b1f..2dcb312898 100644
--- a/pkg/js/generated/go/libldap/ldap.go
+++ b/pkg/js/generated/go/libldap/ldap.go
@@ -53,10 +53,12 @@ func init() {
"FilterWorkstationTrustAccount": lib_ldap.FilterWorkstationTrustAccount,
// Objects / Classes
- "ADObject": gojs.GetClassConstructor[lib_ldap.ADObject](&lib_ldap.ADObject{}),
- "Client": lib_ldap.NewClient,
- "Config": gojs.GetClassConstructor[lib_ldap.Config](&lib_ldap.Config{}),
- "Metadata": gojs.GetClassConstructor[lib_ldap.Metadata](&lib_ldap.Metadata{}),
+ "Client": lib_ldap.NewClient,
+ "Config": gojs.GetClassConstructor[lib_ldap.Config](&lib_ldap.Config{}),
+ "LdapAttributes": gojs.GetClassConstructor[lib_ldap.LdapAttributes](&lib_ldap.LdapAttributes{}),
+ "LdapEntry": gojs.GetClassConstructor[lib_ldap.LdapEntry](&lib_ldap.LdapEntry{}),
+ "Metadata": gojs.GetClassConstructor[lib_ldap.Metadata](&lib_ldap.Metadata{}),
+ "SearchResult": gojs.GetClassConstructor[lib_ldap.SearchResult](&lib_ldap.SearchResult{}),
},
).Register()
}
diff --git a/pkg/js/generated/ts/kerberos.ts b/pkg/js/generated/ts/kerberos.ts
index 8f9ce98f99..d0af2d0e7e 100755
--- a/pkg/js/generated/ts/kerberos.ts
+++ b/pkg/js/generated/ts/kerberos.ts
@@ -188,9 +188,9 @@ export class Config {
*/
export interface AuthorizationDataEntry {
- ADType?: number,
-
ADData?: Uint8Array,
+
+ ADType?: number,
}
@@ -200,9 +200,9 @@ export interface AuthorizationDataEntry {
*/
export interface BitString {
- BitLength?: number,
-
Bytes?: Uint8Array,
+
+ BitLength?: number,
}
@@ -212,9 +212,9 @@ export interface BitString {
*/
export interface BitString {
- BitLength?: number,
-
Bytes?: Uint8Array,
+
+ BitLength?: number,
}
@@ -236,19 +236,15 @@ export interface Config {
*/
export interface EncTicketPart {
- AuthTime?: Date,
-
- StartTime?: Date,
-
EndTime?: Date,
RenewTill?: Date,
CRealm?: string,
- CAddr?: HostAddress,
+ AuthTime?: Date,
- AuthorizationData?: AuthorizationDataEntry,
+ StartTime?: Date,
Flags?: BitString,
@@ -257,6 +253,10 @@ export interface EncTicketPart {
CName?: PrincipalName,
Transited?: TransitedEncoding,
+
+ CAddr?: HostAddress,
+
+ AuthorizationData?: AuthorizationDataEntry,
}
@@ -266,11 +266,11 @@ export interface EncTicketPart {
*/
export interface EncryptedData {
+ EType?: number,
+
KVNO?: number,
Cipher?: Uint8Array,
-
- EType?: number,
}
@@ -318,41 +318,33 @@ export interface HostAddress {
*/
export interface LibDefaults {
- NoAddresses?: boolean,
+ CCacheType?: number,
- RealmTryDomains?: number,
+ K5LoginAuthoritative?: boolean,
- DNSLookupKDC?: boolean,
+ Proxiable?: boolean,
- DefaultRealm?: string,
+ RDNS?: boolean,
- SafeChecksumType?: number,
+ K5LoginDirectory?: string,
- VerifyAPReqNofail?: boolean,
+ KDCTimeSync?: number,
- AllowWeakCrypto?: boolean,
+ VerifyAPReqNofail?: boolean,
DefaultTGSEnctypes?: string[],
- DefaultTktEnctypeIDs?: number[],
-
- IgnoreAcceptorHostname?: boolean,
+ DefaultTGSEnctypeIDs?: number[],
- K5LoginAuthoritative?: boolean,
+ DNSCanonicalizeHostname?: boolean,
- PermittedEnctypes?: string[],
+ Forwardable?: boolean,
/**
* time in nanoseconds
*/
- Clockskew?: number,
-
- DNSCanonicalizeHostname?: boolean,
-
- Proxiable?: boolean,
-
- RDNS?: boolean,
+ RenewLifetime?: number,
/**
* time in nanoseconds
@@ -362,37 +354,45 @@ export interface LibDefaults {
DefaultClientKeytabName?: string,
+ DefaultTktEnctypeIDs?: number[],
+
+ DNSLookupRealm?: boolean,
+
+ ExtraAddresses?: Uint8Array,
+
+ DefaultRealm?: string,
+
+ NoAddresses?: boolean,
+
+ PreferredPreauthTypes?: number[],
+
PermittedEnctypeIDs?: number[],
- UDPPreferenceLimit?: number,
+ RealmTryDomains?: number,
- DefaultTGSEnctypeIDs?: number[],
+ DefaultKeytabName?: string,
DefaultTktEnctypes?: string[],
- CCacheType?: number,
-
- DNSLookupRealm?: boolean,
+ DNSLookupKDC?: boolean,
- ExtraAddresses?: Uint8Array,
+ IgnoreAcceptorHostname?: boolean,
- PreferredPreauthTypes?: number[],
+ AllowWeakCrypto?: boolean,
Canonicalize?: boolean,
- Forwardable?: boolean,
-
- K5LoginDirectory?: string,
+ SafeChecksumType?: number,
- KDCTimeSync?: number,
+ UDPPreferenceLimit?: number,
/**
* time in nanoseconds
*/
- RenewLifetime?: number,
+ Clockskew?: number,
- DefaultKeytabName?: string,
+ PermittedEnctypes?: string[],
KDCDefaultOptions?: BitString,
}
@@ -416,6 +416,8 @@ export interface PrincipalName {
*/
export interface Realm {
+ Realm?: string,
+
AdminServer?: string[],
DefaultDomain?: string,
@@ -425,8 +427,6 @@ export interface Realm {
KPasswdServer?: string[],
MasterKDC?: string[],
-
- Realm?: string,
}
diff --git a/pkg/js/generated/ts/ldap.ts b/pkg/js/generated/ts/ldap.ts
index daca4f86b9..02a10075b9 100755
--- a/pkg/js/generated/ts/ldap.ts
+++ b/pkg/js/generated/ts/ldap.ts
@@ -209,8 +209,8 @@ export class Client {
* log(to_json(users));
* ```
*/
- public FindADObjects(filter: string): ADObject[] {
- return [];
+ public FindADObjects(filter: string): SearchResult | null {
+ return null;
}
@@ -225,8 +225,8 @@ export class Client {
* log(to_json(users));
* ```
*/
- public GetADUsers(): ADObject[] {
- return [];
+ public GetADUsers(): SearchResult | null {
+ return null;
}
@@ -241,8 +241,8 @@ export class Client {
* log(to_json(users));
* ```
*/
- public GetADActiveUsers(): ADObject[] {
- return [];
+ public GetADActiveUsers(): SearchResult | null {
+ return null;
}
@@ -257,8 +257,8 @@ export class Client {
* log(to_json(users));
* ```
*/
- public GetADUserWithNeverExpiringPasswords(): ADObject[] {
- return [];
+ public GetADUserWithNeverExpiringPasswords(): SearchResult | null {
+ return null;
}
@@ -273,8 +273,8 @@ export class Client {
* log(to_json(users));
* ```
*/
- public GetADUserTrustedForDelegation(): ADObject[] {
- return [];
+ public GetADUserTrustedForDelegation(): SearchResult | null {
+ return null;
}
@@ -289,8 +289,8 @@ export class Client {
* log(to_json(users));
* ```
*/
- public GetADUserWithPasswordNotRequired(): ADObject[] {
- return [];
+ public GetADUserWithPasswordNotRequired(): SearchResult | null {
+ return null;
}
@@ -305,8 +305,8 @@ export class Client {
* log(to_json(groups));
* ```
*/
- public GetADGroups(): ADObject[] {
- return [];
+ public GetADGroups(): SearchResult | null {
+ return null;
}
@@ -321,8 +321,8 @@ export class Client {
* log(to_json(dcs));
* ```
*/
- public GetADDCList(): ADObject[] {
- return [];
+ public GetADDCList(): SearchResult | null {
+ return null;
}
@@ -337,8 +337,8 @@ export class Client {
* log(to_json(admins));
* ```
*/
- public GetADAdmins(): ADObject[] {
- return [];
+ public GetADAdmins(): SearchResult | null {
+ return null;
}
@@ -353,8 +353,8 @@ export class Client {
* log(to_json(kerberoastable));
* ```
*/
- public GetADUserKerberoastable(): ADObject[] {
- return [];
+ public GetADUserKerberoastable(): SearchResult | null {
+ return null;
}
@@ -369,8 +369,8 @@ export class Client {
* log(to_json(AsRepRoastable));
* ```
*/
- public GetADUserAsRepRoastable(): ADObject[] {
- return [];
+ public GetADUserAsRepRoastable(): SearchResult | null {
+ return null;
}
@@ -428,8 +428,8 @@ export class Client {
* const results = client.Search('(objectClass=*)', 'cn', 'mail');
* ```
*/
- public Search(filter: string, attributes: any): Record[] {
- return [];
+ public Search(filter: string, attributes: any): SearchResult | null {
+ return null;
}
@@ -481,33 +481,6 @@ export class Client {
-/**
- * ADObject represents an Active Directory object
- * @example
- * ```javascript
- * const ldap = require('nuclei/ldap');
- * const client = new ldap.Client('ldap://ldap.example.com', 'acme.com');
- * const users = client.GetADUsers();
- * log(to_json(users));
- * ```
- */
-export interface ADObject {
-
- DistinguishedName?: string,
-
- SAMAccountName?: string,
-
- PWDLastSet?: string,
-
- LastLogon?: string,
-
- MemberOf?: string[],
-
- ServicePrincipalName?: string[],
-}
-
-
-
/**
* Config is extra configuration for the ldap client
* @example
@@ -535,27 +508,186 @@ export interface Config {
/**
- * Entry Interface
+ * LdapAttributes represents all LDAP attributes of a particular
+ * ldap entry
*/
-export interface Entry {
+export interface LdapAttributes {
- DN?: string,
+ /**
+ * CurrentTime contains current time
+ */
+
+ CurrentTime?: string[],
+
+ /**
+ * SubschemaSubentry contains subschema subentry
+ */
+
+ SubschemaSubentry?: string[],
+
+ /**
+ * DsServiceName contains ds service name
+ */
+
+ DsServiceName?: string[],
+
+ /**
+ * NamingContexts contains naming contexts
+ */
+
+ NamingContexts?: string[],
+
+ /**
+ * DefaultNamingContext contains default naming context
+ */
+
+ DefaultNamingContext?: string[],
+
+ /**
+ * SchemaNamingContext contains schema naming context
+ */
+
+ SchemaNamingContext?: string[],
+
+ /**
+ * ConfigurationNamingContext contains configuration naming context
+ */
+
+ ConfigurationNamingContext?: string[],
+
+ /**
+ * RootDomainNamingContext contains root domain naming context
+ */
+
+ RootDomainNamingContext?: string[],
+
+ /**
+ * SupportedLDAPVersion contains supported LDAP version
+ */
+
+ SupportedLDAPVersion?: string[],
+
+ /**
+ * HighestCommittedUSN contains highest committed USN
+ */
+
+ HighestCommittedUSN?: string[],
+
+ /**
+ * SupportedSASLMechanisms contains supported SASL mechanisms
+ */
+
+ SupportedSASLMechanisms?: string[],
+
+ /**
+ * DnsHostName contains DNS host name
+ */
+
+ DnsHostName?: string[],
- Attributes?: EntryAttribute,
+ /**
+ * LdapServiceName contains LDAP service name
+ */
+
+ LdapServiceName?: string[],
+
+ /**
+ * ServerName contains server name
+ */
+
+ ServerName?: string[],
+
+ /**
+ * IsSynchronized contains is synchronized
+ */
+
+ IsSynchronized?: string[],
+
+ /**
+ * IsGlobalCatalogReady contains is global catalog ready
+ */
+
+ IsGlobalCatalogReady?: string[],
+
+ /**
+ * DomainFunctionality contains domain functionality
+ */
+
+ DomainFunctionality?: string[],
+
+ /**
+ * ForestFunctionality contains forest functionality
+ */
+
+ ForestFunctionality?: string[],
+
+ /**
+ * DomainControllerFunctionality contains domain controller functionality
+ */
+
+ DomainControllerFunctionality?: string[],
+
+ /**
+ * DistinguishedName contains the distinguished name
+ */
+
+ DistinguishedName?: string[],
+
+ /**
+ * SAMAccountName contains the SAM account name
+ */
+
+ SAMAccountName?: string[],
+
+ /**
+ * PWDLastSet contains the password last set time
+ */
+
+ PWDLastSet?: string[],
+
+ /**
+ * LastLogon contains the last logon time
+ */
+
+ LastLogon?: string[],
+
+ /**
+ * MemberOf contains the groups the entry is a member of
+ */
+
+ MemberOf?: string[],
+
+ /**
+ * ServicePrincipalName contains the service principal names
+ */
+
+ ServicePrincipalName?: string[],
+
+ /**
+ * Extra contains other extra fields which might be present
+ */
+
+ Extra?: Record,
}
/**
- * EntryAttribute Interface
+ * LdapEntry represents a single LDAP entry
*/
-export interface EntryAttribute {
+export interface LdapEntry {
- Name?: string,
+ /**
+ * DN contains distinguished name
+ */
- Values?: string[],
+ DN?: string,
- ByteValues?: Uint8Array,
+ /**
+ * Attributes contains list of attributes
+ */
+
+ Attributes?: LdapAttributes,
}
@@ -584,12 +716,32 @@ export interface Metadata {
/**
- * SearchResult Interface
+ * SearchResult contains search result of any / all ldap search request
+ * @example
+ * ```javascript
+ * const ldap = require('nuclei/ldap');
+ * const client = new ldap.Client('ldap://ldap.example.com', 'acme.com');
+ * const results = client.Search('(objectinterface=*)', 'cn', 'mail');
+ * ```
*/
export interface SearchResult {
+ /**
+ * Referrals contains list of referrals
+ */
+
Referrals?: string[],
- Entries?: Entry,
+ /**
+ * Controls contains list of controls
+ */
+
+ Controls?: string[],
+
+ /**
+ * Entries contains list of entries
+ */
+
+ Entries?: LdapEntry[],
}
diff --git a/pkg/js/generated/ts/mssql.ts b/pkg/js/generated/ts/mssql.ts
index bff2edf8b5..947aa0e9ed 100755
--- a/pkg/js/generated/ts/mssql.ts
+++ b/pkg/js/generated/ts/mssql.ts
@@ -2,7 +2,7 @@
/**
* Client is a client for MS SQL database.
- * Internally client uses denisenkom/go-mssqldb driver.
+ * Internally client uses microsoft/go-mssqldb driver.
* @example
* ```javascript
* const mssql = require('nuclei/mssql');
diff --git a/pkg/js/generated/ts/rdp.ts b/pkg/js/generated/ts/rdp.ts
index 77dfeafb13..7858b78319 100755
--- a/pkg/js/generated/ts/rdp.ts
+++ b/pkg/js/generated/ts/rdp.ts
@@ -78,8 +78,6 @@ export interface IsRDPResponse {
*/
export interface ServiceRDP {
- DNSDomainName?: string,
-
ForestName?: string,
OSFingerprint?: string,
@@ -93,5 +91,7 @@ export interface ServiceRDP {
NetBIOSDomainName?: string,
DNSComputerName?: string,
+
+ DNSDomainName?: string,
}
diff --git a/pkg/js/generated/ts/smb.ts b/pkg/js/generated/ts/smb.ts
index b0ca997ac5..c9b769cb5d 100755
--- a/pkg/js/generated/ts/smb.ts
+++ b/pkg/js/generated/ts/smb.ts
@@ -137,6 +137,10 @@ export interface NegotiationLog {
*/
export interface SMBCapabilities {
+ DFSSupport?: boolean,
+
+ Leasing?: boolean,
+
LargeMTU?: boolean,
MultiChan?: boolean,
@@ -146,10 +150,6 @@ export interface SMBCapabilities {
DirLeasing?: boolean,
Encryption?: boolean,
-
- DFSSupport?: boolean,
-
- Leasing?: boolean,
}
@@ -159,16 +159,16 @@ export interface SMBCapabilities {
*/
export interface SMBLog {
- SupportV1?: boolean,
-
- NativeOs?: string,
-
NTLM?: string,
GroupName?: string,
HasNTLM?: boolean,
+ SupportV1?: boolean,
+
+ NativeOs?: string,
+
Version?: SMBVersions,
Capabilities?: SMBCapabilities,
@@ -185,13 +185,13 @@ export interface SMBLog {
*/
export interface SMBVersions {
- VerString?: string,
-
Major?: number,
Minor?: number,
Revision?: number,
+
+ VerString?: string,
}
@@ -225,12 +225,12 @@ export interface ServiceSMB {
*/
export interface SessionSetupLog {
- NegotiateFlags?: number,
-
SetupFlags?: number,
TargetName?: string,
+ NegotiateFlags?: number,
+
HeaderLog?: HeaderLog,
}
diff --git a/pkg/js/generated/ts/ssh.ts b/pkg/js/generated/ts/ssh.ts
index 21dacf4054..058df03c5b 100755
--- a/pkg/js/generated/ts/ssh.ts
+++ b/pkg/js/generated/ts/ssh.ts
@@ -133,9 +133,9 @@ export interface Algorithms {
HostKey?: string,
- R?: DirectionAlgorithms,
-
W?: DirectionAlgorithms,
+
+ R?: DirectionAlgorithms,
}
@@ -159,13 +159,13 @@ export interface DirectionAlgorithms {
*/
export interface EndpointId {
- Raw?: string,
-
- ProtoVersion?: string,
-
SoftwareVersion?: string,
Comment?: string,
+
+ Raw?: string,
+
+ ProtoVersion?: string,
}
@@ -197,34 +197,34 @@ export interface HandshakeLog {
*/
export interface KexInitMsg {
- Reserved?: number,
-
- /**
- * fixed size array of length: [16]
- */
-
- Cookie?: Uint8Array,
+ KexAlgos?: string[],
CiphersClientServer?: string[],
- MACsClientServer?: string[],
-
MACsServerClient?: string[],
+ LanguagesClientServer?: string[],
+
+ CompressionClientServer?: string[],
+
CompressionServerClient?: string[],
- LanguagesClientServer?: string[],
+ Reserved?: number,
- FirstKexFollows?: boolean,
+ MACsClientServer?: string[],
- KexAlgos?: string[],
+ /**
+ * fixed size array of length: [16]
+ */
- CiphersServerClient?: string[],
+ Cookie?: Uint8Array,
- CompressionClientServer?: string[],
+ ServerHostKeyAlgos?: string[],
+
+ CiphersServerClient?: string[],
LanguagesServerClient?: string[],
- ServerHostKeyAlgos?: string[],
+ FirstKexFollows?: boolean,
}
diff --git a/pkg/js/global/scripts.go b/pkg/js/global/scripts.go
index 507446c8b3..3fa7abe4e6 100644
--- a/pkg/js/global/scripts.go
+++ b/pkg/js/global/scripts.go
@@ -2,6 +2,7 @@ package global
import (
"bytes"
+ "context"
"embed"
"math/rand"
"net"
@@ -14,6 +15,14 @@ import (
"github.com/dop251/goja"
"github.com/logrusorgru/aurora"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/js/gojs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/utils/errkit"
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -109,13 +118,43 @@ func initBuiltInFunc(runtime *goja.Runtime) {
Signatures: []string{
"isPortOpen(host string, port string, [timeout int]) bool",
},
- Description: "isPortOpen checks if given port is open on host. timeout is optional and defaults to 5 seconds",
+ Description: "isPortOpen checks if given TCP port is open on host. timeout is optional and defaults to 5 seconds",
FuncDecl: func(host string, port string, timeout ...int) (bool, error) {
- timeoutInSec := 5
+ ctx := context.Background()
if len(timeout) > 0 {
- timeoutInSec = timeout[0]
+ var cancel context.CancelFunc
+ ctx, cancel = context.WithTimeout(ctx, time.Duration(timeout[0])*time.Second)
+ defer cancel()
}
- conn, err := net.DialTimeout("tcp", net.JoinHostPort(host, port), time.Duration(timeoutInSec)*time.Second)
+ if host == "" || port == "" {
+ return false, errkit.New("isPortOpen: host or port is empty")
+ }
+ conn, err := protocolstate.Dialer.Dial(ctx, "tcp", net.JoinHostPort(host, port))
+ if err != nil {
+ return false, err
+ }
+ _ = conn.Close()
+ return true, nil
+ },
+ })
+
+ _ = gojs.RegisterFuncWithSignature(runtime, gojs.FuncOpts{
+ Name: "isUDPPortOpen",
+ Signatures: []string{
+ "isUDPPortOpen(host string, port string, [timeout int]) bool",
+ },
+ Description: "isUDPPortOpen checks if the given UDP port is open on the host. Timeout is optional and defaults to 5 seconds.",
+ FuncDecl: func(host string, port string, timeout ...int) (bool, error) {
+ ctx := context.Background()
+ if len(timeout) > 0 {
+ var cancel context.CancelFunc
+ ctx, cancel = context.WithTimeout(ctx, time.Duration(timeout[0])*time.Second)
+ defer cancel()
+ }
+ if host == "" || port == "" {
+ return false, errkit.New("isPortOpen: host or port is empty")
+ }
+ conn, err := protocolstate.Dialer.Dial(ctx, "udp", net.JoinHostPort(host, port))
if err != nil {
return false, err
}
diff --git a/pkg/js/libs/ldap/adenum.go b/pkg/js/libs/ldap/adenum.go
index 9aea98be97..849e4d6a15 100644
--- a/pkg/js/libs/ldap/adenum.go
+++ b/pkg/js/libs/ldap/adenum.go
@@ -74,25 +74,6 @@ func NegativeFilter(filter string) string {
return fmt.Sprintf("(!%s)", filter)
}
-type (
- // ADObject represents an Active Directory object
- // @example
- // ```javascript
- // const ldap = require('nuclei/ldap');
- // const client = new ldap.Client('ldap://ldap.example.com', 'acme.com');
- // const users = client.GetADUsers();
- // log(to_json(users));
- // ```
- ADObject struct {
- DistinguishedName string
- SAMAccountName string
- PWDLastSet string
- LastLogon string
- MemberOf []string
- ServicePrincipalName []string
- }
-)
-
// FindADObjects finds AD objects based on a filter
// and returns them as a list of ADObject
// @example
@@ -102,7 +83,7 @@ type (
// const users = client.FindADObjects(ldap.FilterIsPerson);
// log(to_json(users));
// ```
-func (c *Client) FindADObjects(filter string) []ADObject {
+func (c *Client) FindADObjects(filter string) SearchResult {
c.nj.Require(c.conn != nil, "no existing connection")
sr := ldap.NewSearchRequest(
c.BaseDN, ldap.ScopeWholeSubtree,
@@ -121,19 +102,7 @@ func (c *Client) FindADObjects(filter string) []ADObject {
res, err := c.conn.Search(sr)
c.nj.HandleError(err, "ldap search request failed")
-
- var objects []ADObject
- for _, obj := range res.Entries {
- objects = append(objects, ADObject{
- DistinguishedName: obj.GetAttributeValue("distinguishedName"),
- SAMAccountName: obj.GetAttributeValue("sAMAccountName"),
- PWDLastSet: DecodeADTimestamp(obj.GetAttributeValue("pwdLastSet")),
- LastLogon: DecodeADTimestamp(obj.GetAttributeValue("lastLogon")),
- MemberOf: obj.GetAttributeValues("memberOf"),
- ServicePrincipalName: obj.GetAttributeValues("servicePrincipalName"),
- })
- }
- return objects
+ return *getSearchResult(res)
}
// GetADUsers returns all AD users
@@ -145,7 +114,7 @@ func (c *Client) FindADObjects(filter string) []ADObject {
// const users = client.GetADUsers();
// log(to_json(users));
// ```
-func (c *Client) GetADUsers() []ADObject {
+func (c *Client) GetADUsers() SearchResult {
return c.FindADObjects(FilterIsPerson)
}
@@ -158,7 +127,7 @@ func (c *Client) GetADUsers() []ADObject {
// const users = client.GetADActiveUsers();
// log(to_json(users));
// ```
-func (c *Client) GetADActiveUsers() []ADObject {
+func (c *Client) GetADActiveUsers() SearchResult {
return c.FindADObjects(JoinFilters(FilterIsPerson, FilterAccountEnabled))
}
@@ -171,7 +140,7 @@ func (c *Client) GetADActiveUsers() []ADObject {
// const users = client.GetADUserWithNeverExpiringPasswords();
// log(to_json(users));
// ```
-func (c *Client) GetADUserWithNeverExpiringPasswords() []ADObject {
+func (c *Client) GetADUserWithNeverExpiringPasswords() SearchResult {
return c.FindADObjects(JoinFilters(FilterIsPerson, FilterDontExpirePassword))
}
@@ -184,7 +153,7 @@ func (c *Client) GetADUserWithNeverExpiringPasswords() []ADObject {
// const users = client.GetADUserTrustedForDelegation();
// log(to_json(users));
// ```
-func (c *Client) GetADUserTrustedForDelegation() []ADObject {
+func (c *Client) GetADUserTrustedForDelegation() SearchResult {
return c.FindADObjects(JoinFilters(FilterIsPerson, FilterTrustedForDelegation))
}
@@ -197,7 +166,7 @@ func (c *Client) GetADUserTrustedForDelegation() []ADObject {
// const users = client.GetADUserWithPasswordNotRequired();
// log(to_json(users));
// ```
-func (c *Client) GetADUserWithPasswordNotRequired() []ADObject {
+func (c *Client) GetADUserWithPasswordNotRequired() SearchResult {
return c.FindADObjects(JoinFilters(FilterIsPerson, FilterPasswordNotRequired))
}
@@ -210,7 +179,7 @@ func (c *Client) GetADUserWithPasswordNotRequired() []ADObject {
// const groups = client.GetADGroups();
// log(to_json(groups));
// ```
-func (c *Client) GetADGroups() []ADObject {
+func (c *Client) GetADGroups() SearchResult {
return c.FindADObjects(FilterIsGroup)
}
@@ -223,7 +192,7 @@ func (c *Client) GetADGroups() []ADObject {
// const dcs = client.GetADDCList();
// log(to_json(dcs));
// ```
-func (c *Client) GetADDCList() []ADObject {
+func (c *Client) GetADDCList() SearchResult {
return c.FindADObjects(JoinFilters(FilterIsComputer, FilterAccountEnabled, FilterServerTrustAccount))
}
@@ -236,7 +205,7 @@ func (c *Client) GetADDCList() []ADObject {
// const admins = client.GetADAdmins();
// log(to_json(admins));
// ```
-func (c *Client) GetADAdmins() []ADObject {
+func (c *Client) GetADAdmins() SearchResult {
return c.FindADObjects(JoinFilters(FilterIsPerson, FilterAccountEnabled, FilterIsAdmin))
}
@@ -249,7 +218,7 @@ func (c *Client) GetADAdmins() []ADObject {
// const kerberoastable = client.GetADUserKerberoastable();
// log(to_json(kerberoastable));
// ```
-func (c *Client) GetADUserKerberoastable() []ADObject {
+func (c *Client) GetADUserKerberoastable() SearchResult {
return c.FindADObjects(JoinFilters(FilterIsPerson, FilterAccountEnabled, FilterHasServicePrincipalName))
}
@@ -262,7 +231,7 @@ func (c *Client) GetADUserKerberoastable() []ADObject {
// const AsRepRoastable = client.GetADUserAsRepRoastable();
// log(to_json(AsRepRoastable));
// ```
-func (c *Client) GetADUserAsRepRoastable() []ADObject {
+func (c *Client) GetADUserAsRepRoastable() SearchResult {
return c.FindADObjects(JoinFilters(FilterIsPerson, FilterDontRequirePreauth))
}
@@ -276,7 +245,16 @@ func (c *Client) GetADUserAsRepRoastable() []ADObject {
// ```
func (c *Client) GetADDomainSID() string {
r := c.Search(FilterServerTrustAccount, "objectSid")
- c.nj.Require(len(r) > 0, "no result from GetADDomainSID query")
- c.nj.Require(len(r[0]["objectSid"]) > 0, "could not grab DomainSID")
- return DecodeSID(r[0]["objectSid"][0])
+ c.nj.Require(len(r.Entries) > 0, "no result from GetADDomainSID query")
+ for _, entry := range r.Entries {
+ if sid, ok := entry.Attributes.Extra["objectSid"]; ok {
+ if sid, ok := sid.([]string); ok {
+ return DecodeSID(sid[0])
+ } else {
+ c.nj.HandleError(fmt.Errorf("invalid objectSid type: %T", entry.Attributes.Extra["objectSid"]), "invalid objectSid type")
+ }
+ }
+ }
+ c.nj.HandleError(fmt.Errorf("no objectSid found"), "no objectSid found")
+ return ""
}
diff --git a/pkg/js/libs/ldap/ldap.go b/pkg/js/libs/ldap/ldap.go
index 6bc1aff937..e022e4a05f 100644
--- a/pkg/js/libs/ldap/ldap.go
+++ b/pkg/js/libs/ldap/ldap.go
@@ -155,7 +155,7 @@ func NewClient(call goja.ConstructorCall, runtime *goja.Runtime) *goja.Object {
// const client = new ldap.Client('ldap://ldap.example.com', 'acme.com');
// client.Authenticate('user', 'password');
// ```
-func (c *Client) Authenticate(username, password string) {
+func (c *Client) Authenticate(username, password string) bool {
c.nj.Require(c.conn != nil, "no existing connection")
if c.BaseDN == "" {
c.BaseDN = fmt.Sprintf("dc=%s", strings.Join(strings.Split(c.Realm, "."), ",dc="))
@@ -163,19 +163,21 @@ func (c *Client) Authenticate(username, password string) {
if err := c.conn.NTLMBind(c.Realm, username, password); err == nil {
// if bind with NTLMBind(), there is nothing
// else to do, you are authenticated
- return
+ return true
}
+ var err error
switch password {
case "":
- if err := c.conn.UnauthenticatedBind(username); err != nil {
+ if err = c.conn.UnauthenticatedBind(username); err != nil {
c.nj.ThrowError(err)
}
default:
- if err := c.conn.Bind(username, password); err != nil {
+ if err = c.conn.Bind(username, password); err != nil {
c.nj.ThrowError(err)
}
}
+ return err == nil
}
// AuthenticateWithNTLMHash authenticates with the ldap server using the given username and NTLM hash
@@ -185,14 +187,16 @@ func (c *Client) Authenticate(username, password string) {
// const client = new ldap.Client('ldap://ldap.example.com', 'acme.com');
// client.AuthenticateWithNTLMHash('pdtm', 'hash');
// ```
-func (c *Client) AuthenticateWithNTLMHash(username, hash string) {
+func (c *Client) AuthenticateWithNTLMHash(username, hash string) bool {
c.nj.Require(c.conn != nil, "no existing connection")
if c.BaseDN == "" {
c.BaseDN = fmt.Sprintf("dc=%s", strings.Join(strings.Split(c.Realm, "."), ",dc="))
}
- if err := c.conn.NTLMBindWithHash(c.Realm, username, hash); err != nil {
+ var err error
+ if err = c.conn.NTLMBindWithHash(c.Realm, username, hash); err != nil {
c.nj.ThrowError(err)
}
+ return err == nil
}
// Search accepts whatever filter and returns a list of maps having provided attributes
@@ -203,38 +207,24 @@ func (c *Client) AuthenticateWithNTLMHash(username, hash string) {
// const client = new ldap.Client('ldap://ldap.example.com', 'acme.com');
// const results = client.Search('(objectClass=*)', 'cn', 'mail');
// ```
-func (c *Client) Search(filter string, attributes ...string) []map[string][]string {
+func (c *Client) Search(filter string, attributes ...string) SearchResult {
c.nj.Require(c.conn != nil, "no existing connection")
+ c.nj.Require(c.BaseDN != "", "base dn cannot be empty")
+ c.nj.Require(len(attributes) > 0, "attributes cannot be empty")
res, err := c.conn.Search(
ldap.NewSearchRequest(
- c.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
- 0, 0, false, filter, attributes, nil,
+ "",
+ ldap.ScopeWholeSubtree,
+ ldap.NeverDerefAliases,
+ 0, 0, false,
+ filter,
+ attributes,
+ nil,
),
)
c.nj.HandleError(err, "ldap search request failed")
- if len(res.Entries) == 0 {
- // return empty list
- return nil
- }
-
- // convert ldap.Entry to []map[string][]string
- var out []map[string][]string
- for _, r := range res.Entries {
- app := make(map[string][]string)
- empty := true
- for _, a := range attributes {
- v := r.GetAttributeValues(a)
- if len(v) > 0 {
- app[a] = v
- empty = false
- }
- }
- if !empty {
- out = append(out, app)
- }
- }
- return out
+ return *getSearchResult(res)
}
// AdvancedSearch accepts all values of search request type and return Ldap Entry
@@ -250,7 +240,7 @@ func (c *Client) AdvancedSearch(
TypesOnly bool,
Filter string,
Attributes []string,
- Controls []ldap.Control) ldap.SearchResult {
+ Controls []ldap.Control) SearchResult {
c.nj.Require(c.conn != nil, "no existing connection")
if c.BaseDN == "" {
c.BaseDN = fmt.Sprintf("dc=%s", strings.Join(strings.Split(c.Realm, "."), ",dc="))
@@ -259,7 +249,7 @@ func (c *Client) AdvancedSearch(
res, err := c.conn.Search(req)
c.nj.HandleError(err, "ldap search request failed")
c.nj.Require(res != nil, "ldap search request failed got nil response")
- return *res
+ return *getSearchResult(res)
}
type (
@@ -293,6 +283,7 @@ func (c *Client) CollectMetadata() Metadata {
}
metadata.BaseDN = c.BaseDN
+ // Use scope as Base since Root DSE doesn't have subentries
srMetadata := ldap.NewSearchRequest(
"",
ldap.ScopeBaseObject,
diff --git a/pkg/js/libs/ldap/utils.go b/pkg/js/libs/ldap/utils.go
index 96a243bbc7..bcff41dc49 100644
--- a/pkg/js/libs/ldap/utils.go
+++ b/pkg/js/libs/ldap/utils.go
@@ -5,8 +5,189 @@ import (
"strconv"
"strings"
"time"
+
+ "github.com/go-ldap/ldap/v3"
)
+type (
+ // SearchResult contains search result of any / all ldap search request
+ // @example
+ // ```javascript
+ // const ldap = require('nuclei/ldap');
+ // const client = new ldap.Client('ldap://ldap.example.com', 'acme.com');
+ // const results = client.Search('(objectClass=*)', 'cn', 'mail');
+ // ```
+ SearchResult struct {
+ // Referrals contains list of referrals
+ Referrals []string `json:"referrals"`
+ // Controls contains list of controls
+ Controls []string `json:"controls"`
+ // Entries contains list of entries
+ Entries []LdapEntry `json:"entries"`
+ }
+
+ // LdapEntry represents a single LDAP entry
+ LdapEntry struct {
+ // DN contains distinguished name
+ DN string `json:"dn"`
+ // Attributes contains list of attributes
+ Attributes LdapAttributes `json:"attributes"`
+ }
+
+ // LdapAttributes represents all LDAP attributes of a particular
+ // ldap entry
+ LdapAttributes struct {
+ // CurrentTime contains current time
+ CurrentTime []string `json:"currentTime,omitempty"`
+ // SubschemaSubentry contains subschema subentry
+ SubschemaSubentry []string `json:"subschemaSubentry,omitempty"`
+ // DsServiceName contains ds service name
+ DsServiceName []string `json:"dsServiceName,omitempty"`
+ // NamingContexts contains naming contexts
+ NamingContexts []string `json:"namingContexts,omitempty"`
+ // DefaultNamingContext contains default naming context
+ DefaultNamingContext []string `json:"defaultNamingContext,omitempty"`
+ // SchemaNamingContext contains schema naming context
+ SchemaNamingContext []string `json:"schemaNamingContext,omitempty"`
+ // ConfigurationNamingContext contains configuration naming context
+ ConfigurationNamingContext []string `json:"configurationNamingContext,omitempty"`
+ // RootDomainNamingContext contains root domain naming context
+ RootDomainNamingContext []string `json:"rootDomainNamingContext,omitempty"`
+ // SupportedLDAPVersion contains supported LDAP version
+ SupportedLDAPVersion []string `json:"supportedLDAPVersion,omitempty"`
+ // HighestCommittedUSN contains highest committed USN
+ HighestCommittedUSN []string `json:"highestCommittedUSN,omitempty"`
+ // SupportedSASLMechanisms contains supported SASL mechanisms
+ SupportedSASLMechanisms []string `json:"supportedSASLMechanisms,omitempty"`
+ // DnsHostName contains DNS host name
+ DnsHostName []string `json:"dnsHostName,omitempty"`
+ // LdapServiceName contains LDAP service name
+ LdapServiceName []string `json:"ldapServiceName,omitempty"`
+ // ServerName contains server name
+ ServerName []string `json:"serverName,omitempty"`
+ // IsSynchronized contains is synchronized
+ IsSynchronized []string `json:"isSynchronized,omitempty"`
+ // IsGlobalCatalogReady contains is global catalog ready
+ IsGlobalCatalogReady []string `json:"isGlobalCatalogReady,omitempty"`
+ // DomainFunctionality contains domain functionality
+ DomainFunctionality []string `json:"domainFunctionality,omitempty"`
+ // ForestFunctionality contains forest functionality
+ ForestFunctionality []string `json:"forestFunctionality,omitempty"`
+ // DomainControllerFunctionality contains domain controller functionality
+ DomainControllerFunctionality []string `json:"domainControllerFunctionality,omitempty"`
+ // DistinguishedName contains the distinguished name
+ DistinguishedName []string `json:"distinguishedName,omitempty"`
+ // SAMAccountName contains the SAM account name
+ SAMAccountName []string `json:"sAMAccountName,omitempty"`
+ // PWDLastSet contains the password last set time
+ PWDLastSet []string `json:"pwdLastSet,omitempty"`
+ // LastLogon contains the last logon time
+ LastLogon []string `json:"lastLogon,omitempty"`
+ // MemberOf contains the groups the entry is a member of
+ MemberOf []string `json:"memberOf,omitempty"`
+ // ServicePrincipalName contains the service principal names
+ ServicePrincipalName []string `json:"servicePrincipalName,omitempty"`
+ // Extra contains other extra fields which might be present
+ Extra map[string]any `json:"extra,omitempty"`
+ }
+)
+
+// getSearchResult converts a ldap.SearchResult to a SearchResult
+func getSearchResult(sr *ldap.SearchResult) *SearchResult {
+ t := &SearchResult{
+ Referrals: []string{},
+ Controls: []string{},
+ Entries: []LdapEntry{},
+ }
+ // add referrals
+ t.Referrals = append(t.Referrals, sr.Referrals...)
+ // add controls
+ for _, ctrl := range sr.Controls {
+ t.Controls = append(t.Controls, ctrl.String())
+ }
+ // add entries
+ for _, entry := range sr.Entries {
+ t.Entries = append(t.Entries, parseLdapEntry(entry))
+ }
+ return t
+}
+
+func parseLdapEntry(entry *ldap.Entry) LdapEntry {
+ e := LdapEntry{
+ DN: entry.DN,
+ }
+ attrs := LdapAttributes{
+ Extra: make(map[string]any),
+ }
+ for _, attr := range entry.Attributes {
+ switch attr.Name {
+ case "currentTime":
+ attrs.CurrentTime = decodeTimestamps(attr.Values)
+ case "subschemaSubentry":
+ attrs.SubschemaSubentry = attr.Values
+ case "dsServiceName":
+ attrs.DsServiceName = attr.Values
+ case "namingContexts":
+ attrs.NamingContexts = attr.Values
+ case "defaultNamingContext":
+ attrs.DefaultNamingContext = attr.Values
+ case "schemaNamingContext":
+ attrs.SchemaNamingContext = attr.Values
+ case "configurationNamingContext":
+ attrs.ConfigurationNamingContext = attr.Values
+ case "rootDomainNamingContext":
+ attrs.RootDomainNamingContext = attr.Values
+ case "supportedLDAPVersion":
+ attrs.SupportedLDAPVersion = attr.Values
+ case "highestCommittedUSN":
+ attrs.HighestCommittedUSN = attr.Values
+ case "supportedSASLMechanisms":
+ attrs.SupportedSASLMechanisms = attr.Values
+ case "dnsHostName":
+ attrs.DnsHostName = attr.Values
+ case "ldapServiceName":
+ attrs.LdapServiceName = attr.Values
+ case "serverName":
+ attrs.ServerName = attr.Values
+ case "isSynchronized":
+ attrs.IsSynchronized = attr.Values
+ case "isGlobalCatalogReady":
+ attrs.IsGlobalCatalogReady = attr.Values
+ case "domainFunctionality":
+ attrs.DomainFunctionality = attr.Values
+ case "forestFunctionality":
+ attrs.ForestFunctionality = attr.Values
+ case "domainControllerFunctionality":
+ attrs.DomainControllerFunctionality = attr.Values
+ case "distinguishedName":
+ attrs.DistinguishedName = attr.Values
+ case "sAMAccountName":
+ attrs.SAMAccountName = attr.Values
+ case "pwdLastSet":
+ attrs.PWDLastSet = decodeTimestamps(attr.Values)
+ case "lastLogon":
+ attrs.LastLogon = decodeTimestamps(attr.Values)
+ case "memberOf":
+ attrs.MemberOf = attr.Values
+ case "servicePrincipalName":
+ attrs.ServicePrincipalName = attr.Values
+ default:
+ attrs.Extra[attr.Name] = attr.Values
+ }
+ }
+ e.Attributes = attrs
+ return e
+}
+
+// decodeTimestamps decodes multiple timestamps
+func decodeTimestamps(timestamps []string) []string {
+ res := []string{}
+ for _, timestamp := range timestamps {
+ res = append(res, DecodeADTimestamp(timestamp))
+ }
+ return res
+}
+
// DecodeSID decodes a SID string
// @example
// ```javascript
diff --git a/pkg/js/libs/mssql/memo.mssql.go b/pkg/js/libs/mssql/memo.mssql.go
index 8f9ad11708..7a50fff932 100755
--- a/pkg/js/libs/mssql/memo.mssql.go
+++ b/pkg/js/libs/mssql/memo.mssql.go
@@ -5,7 +5,7 @@ import (
"errors"
"fmt"
- _ "github.com/denisenkom/go-mssqldb"
+ _ "github.com/microsoft/go-mssqldb"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/protocolstate"
)
diff --git a/pkg/js/libs/mssql/mssql.go b/pkg/js/libs/mssql/mssql.go
index 55cbe82558..77c109a90b 100644
--- a/pkg/js/libs/mssql/mssql.go
+++ b/pkg/js/libs/mssql/mssql.go
@@ -9,14 +9,18 @@ import (
"strings"
"time"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/protocolstate"
_ "github.com/denisenkom/go-mssqldb"
+=======
+ _ "github.com/microsoft/go-mssqldb"
+>>>>>>> projectdiscovery-main
"github.com/praetorian-inc/fingerprintx/pkg/plugins/services/mssql"
)
type (
// Client is a client for MS SQL database.
- // Internally client uses denisenkom/go-mssqldb driver.
+ // Internally client uses microsoft/go-mssqldb driver.
// @example
// ```javascript
// const mssql = require('nuclei/mssql');
diff --git a/pkg/js/libs/mysql/mysql.go b/pkg/js/libs/mysql/mysql.go
index 6f8b54bde4..fe5cecbf72 100644
--- a/pkg/js/libs/mysql/mysql.go
+++ b/pkg/js/libs/mysql/mysql.go
@@ -36,6 +36,7 @@ type (
// const isMySQL = mysql.IsMySQL('acme.com', 3306);
// ```
func (c *MySQLClient) IsMySQL(host string, port int) (bool, error) {
+ // todo: why this is exposed? Service fingerprint should be automatic
return memoizedisMySQL(host, port)
}
@@ -77,6 +78,16 @@ func (c *MySQLClient) Connect(host string, port int, username, password string)
// host is not valid according to network policy
return false, protocolstate.ErrHostDenied.Msgf(host)
}
+
+ // executing queries implies the remote mysql service
+ ok, err := c.IsMySQL(host, port)
+ if err != nil {
+ return false, err
+ }
+ if !ok {
+ return false, fmt.Errorf("not a mysql service")
+ }
+
dsn, err := BuildDSN(MySQLOptions{
Host: host,
Port: port,
@@ -182,6 +193,16 @@ func (c *MySQLClient) ExecuteQueryWithOpts(opts MySQLOptions, query string) (*ut
// host is not valid according to network policy
return nil, protocolstate.ErrHostDenied.Msgf(opts.Host)
}
+
+ // executing queries implies the remote mysql service
+ ok, err := c.IsMySQL(opts.Host, opts.Port)
+ if err != nil {
+ return nil, err
+ }
+ if !ok {
+ return nil, fmt.Errorf("not a mysql service")
+ }
+
dsn, err := BuildDSN(opts)
if err != nil {
return nil, err
@@ -220,6 +241,15 @@ func (c *MySQLClient) ExecuteQueryWithOpts(opts MySQLOptions, query string) (*ut
// log(to_json(result));
// ```
func (c *MySQLClient) ExecuteQuery(host string, port int, username, password, query string) (*utils.SQLResult, error) {
+ // executing queries implies the remote mysql service
+ ok, err := c.IsMySQL(host, port)
+ if err != nil {
+ return nil, err
+ }
+ if !ok {
+ return nil, fmt.Errorf("not a mysql service")
+ }
+
return c.ExecuteQueryWithOpts(MySQLOptions{
Host: host,
Port: port,
diff --git a/pkg/js/libs/mysql/mysql_private.go b/pkg/js/libs/mysql/mysql_private.go
index aa89df2600..42e76a5579 100644
--- a/pkg/js/libs/mysql/mysql_private.go
+++ b/pkg/js/libs/mysql/mysql_private.go
@@ -53,7 +53,7 @@ func BuildDSN(opts MySQLOptions) (string, error) {
}
target := net.JoinHostPort(opts.Host, fmt.Sprintf("%d", opts.Port))
var dsn strings.Builder
- dsn.WriteString(fmt.Sprintf("%v:%v", url.QueryEscape(opts.Username), url.QueryEscape(opts.Password)))
+ dsn.WriteString(fmt.Sprintf("%v:%v", url.QueryEscape(opts.Username), opts.Password))
dsn.WriteString("@")
dsn.WriteString(fmt.Sprintf("%v(%v)", opts.Protocol, target))
if opts.DbName != "" {
diff --git a/pkg/js/libs/postgres/memo.postgres.go b/pkg/js/libs/postgres/memo.postgres.go
index 2f552492a5..14a2b5f76f 100755
--- a/pkg/js/libs/postgres/memo.postgres.go
+++ b/pkg/js/libs/postgres/memo.postgres.go
@@ -5,7 +5,7 @@ import (
"errors"
"fmt"
- _ "github.com/lib/pq"
+ _ "github.com/projectdiscovery/nuclei/v3/pkg/js/utils/pgwrap"
utils "github.com/Explorer1092/nuclei/v3/pkg/js/utils"
diff --git a/pkg/js/libs/postgres/postgres.go b/pkg/js/libs/postgres/postgres.go
index c4d764cd2e..9a1eac92e6 100644
--- a/pkg/js/libs/postgres/postgres.go
+++ b/pkg/js/libs/postgres/postgres.go
@@ -11,9 +11,15 @@ import (
utils "github.com/Explorer1092/nuclei/v3/pkg/js/utils"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/go-pg/pg"
- _ "github.com/lib/pq"
"github.com/praetorian-inc/fingerprintx/pkg/plugins"
postgres "github.com/praetorian-inc/fingerprintx/pkg/plugins/services/postgresql"
+<<<<<<< HEAD
+=======
+ utils "github.com/projectdiscovery/nuclei/v3/pkg/js/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/js/utils/pgwrap"
+ _ "github.com/projectdiscovery/nuclei/v3/pkg/js/utils/pgwrap"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+>>>>>>> projectdiscovery-main
)
type (
@@ -36,6 +42,7 @@ type (
// const isPostgres = postgres.IsPostgres('acme.com', 5432);
// ```
func (c *PGClient) IsPostgres(host string, port int) (bool, error) {
+ // todo: why this is exposed? Service fingerprint should be automatic
return memoizedisPostgres(host, port)
}
@@ -73,6 +80,13 @@ func isPostgres(host string, port int) (bool, error) {
// const connected = client.Connect('acme.com', 5432, 'username', 'password');
// ```
func (c *PGClient) Connect(host string, port int, username, password string) (bool, error) {
+ ok, err := c.IsPostgres(host, port)
+ if err != nil {
+ return false, err
+ }
+ if !ok {
+ return false, fmt.Errorf("not a postgres service")
+ }
return memoizedconnect(host, port, username, password, "postgres")
}
@@ -87,6 +101,14 @@ func (c *PGClient) Connect(host string, port int, username, password string) (bo
// log(to_json(result));
// ```
func (c *PGClient) ExecuteQuery(host string, port int, username, password, dbName, query string) (*utils.SQLResult, error) {
+ ok, err := c.IsPostgres(host, port)
+ if err != nil {
+ return nil, err
+ }
+ if !ok {
+ return nil, fmt.Errorf("not a postgres service")
+ }
+
return memoizedexecuteQuery(host, port, username, password, dbName, query)
}
@@ -100,10 +122,11 @@ func executeQuery(host string, port int, username string, password string, dbNam
target := net.JoinHostPort(host, fmt.Sprintf("%d", port))
connStr := fmt.Sprintf("postgres://%s:%s@%s/%s?sslmode=disable", username, password, target, dbName)
- db, err := sql.Open("postgres", connStr)
+ db, err := sql.Open(pgwrap.PGWrapDriver, connStr)
if err != nil {
return nil, err
}
+ defer db.Close()
rows, err := db.Query(query)
if err != nil {
@@ -127,6 +150,14 @@ func executeQuery(host string, port int, username string, password string, dbNam
// const connected = client.ConnectWithDB('acme.com', 5432, 'username', 'password', 'dbname');
// ```
func (c *PGClient) ConnectWithDB(host string, port int, username, password, dbName string) (bool, error) {
+ ok, err := c.IsPostgres(host, port)
+ if err != nil {
+ return false, err
+ }
+ if !ok {
+ return false, fmt.Errorf("not a postgres service")
+ }
+
return memoizedconnect(host, port, username, password, dbName)
}
@@ -143,12 +174,19 @@ func connect(host string, port int, username string, password string, dbName str
target := net.JoinHostPort(host, fmt.Sprintf("%d", port))
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+
db := pg.Connect(&pg.Options{
Addr: target,
User: username,
Password: password,
Database: dbName,
- })
+ Dialer: func(network, addr string) (net.Conn, error) {
+ return protocolstate.Dialer.Dial(context.Background(), network, addr)
+ },
+ IdleCheckFrequency: -1,
+ }).WithContext(ctx).WithTimeout(10 * time.Second)
defer db.Close()
_, err := db.Exec("select 1")
diff --git a/pkg/js/utils/nucleijs.go b/pkg/js/utils/nucleijs.go
index d3d4560813..9d9e3f4ece 100644
--- a/pkg/js/utils/nucleijs.go
+++ b/pkg/js/utils/nucleijs.go
@@ -60,12 +60,16 @@ func (j *NucleiJS) HandleError(err error, msg ...string) {
if len(msg) == 0 {
j.ThrowError(err)
}
- j.Throw(fmt.Sprintf("%s: %s", strings.Join(msg, ":"), err.Error()))
+ j.Throw("%s: %s", strings.Join(msg, ":"), err.Error())
}
// Throw throws an error in goja runtime
func (j *NucleiJS) Throw(format string, args ...interface{}) {
- panic(j.runtime().ToValue(fmt.Sprintf(format, args...)))
+ if len(args) > 0 {
+ panic(j.runtime().ToValue(fmt.Sprintf(format, args...)))
+ }
+
+ panic(j.runtime().ToValue(format))
}
// GetArg returns argument at index from goja runtime if not found throws error
@@ -95,7 +99,7 @@ func (j *NucleiJS) GetArgSafe(args []goja.Value, index int, defaultValue any) an
// Require throws an error if expression is false
func (j *NucleiJS) Require(expr bool, msg string) {
if !expr {
- j.Throw(msg)
+ j.Throw("%s", msg)
}
}
diff --git a/pkg/js/utils/pgwrap/pgwrap.go b/pkg/js/utils/pgwrap/pgwrap.go
new file mode 100644
index 0000000000..d1b82f7ab3
--- /dev/null
+++ b/pkg/js/utils/pgwrap/pgwrap.go
@@ -0,0 +1,53 @@
+package pgwrap
+
+import (
+ "context"
+ "database/sql"
+ "database/sql/driver"
+ "net"
+ "time"
+
+ "github.com/lib/pq"
+ "github.com/projectdiscovery/fastdialer/fastdialer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+)
+
+const (
+ PGWrapDriver = "pgwrap"
+)
+
+type pgDial struct {
+ fd *fastdialer.Dialer
+}
+
+func (p *pgDial) Dial(network, address string) (net.Conn, error) {
+ return p.fd.Dial(context.TODO(), network, address)
+}
+
+func (p *pgDial) DialTimeout(network, address string, timeout time.Duration) (net.Conn, error) {
+ ctx, cancel := context.WithTimeoutCause(context.Background(), timeout, fastdialer.ErrDialTimeout)
+ defer cancel()
+ return p.fd.Dial(ctx, network, address)
+}
+
+func (p *pgDial) DialContext(ctx context.Context, network, address string) (net.Conn, error) {
+ return p.fd.Dial(ctx, network, address)
+}
+
+// Unfortunately lib/pq does not provide easy to customize or
+// replace dialer so we need to hijack it by wrapping it in our own
+// driver and register it as postgres driver
+
+// PgDriver is the Postgres database driver.
+type PgDriver struct{}
+
+// Open opens a new connection to the database. name is a connection string.
+// Most users should only use it through database/sql package from the standard
+// library.
+func (d PgDriver) Open(name string) (driver.Conn, error) {
+ return pq.DialOpen(&pgDial{fd: protocolstate.Dialer}, name)
+}
+
+func init() {
+ sql.Register(PGWrapDriver, &PgDriver{})
+}
diff --git a/pkg/keys/key.go b/pkg/keys/key.go
index bb0c900626..eac43f27ae 100644
--- a/pkg/keys/key.go
+++ b/pkg/keys/key.go
@@ -3,5 +3,7 @@ package keys
import _ "embed"
+const PDVerifier = "projectdiscovery/nuclei-templates"
+
//go:embed nuclei.crt
var NucleiCert []byte // public key for verifying digital signature of templates
diff --git a/pkg/loader/workflow/workflow_loader.go b/pkg/loader/workflow/workflow_loader.go
index e7336343ef..13251816e4 100644
--- a/pkg/loader/workflow/workflow_loader.go
+++ b/pkg/loader/workflow/workflow_loader.go
@@ -7,6 +7,21 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/templates"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/parsers/workflow_loader.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/loader/filter"
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader/filter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/loader/workflow/workflow_loader.go
+>>>>>>> projectdiscovery-main
)
type workflowLoader struct {
diff --git a/pkg/model/model.go b/pkg/model/model.go
index 716b15d9d0..48429c9925 100644
--- a/pkg/model/model.go
+++ b/pkg/model/model.go
@@ -1,9 +1,20 @@
package model
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
"github.com/invopop/jsonschema"
+=======
+<<<<<<< HEAD:v2/pkg/model/model.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+=======
+ "github.com/invopop/jsonschema"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/model/model.go
+>>>>>>> projectdiscovery-main
)
type schemaMetadata struct {
diff --git a/pkg/model/model_test.go b/pkg/model/model_test.go
index 55bb73b265..5546386569 100644
--- a/pkg/model/model_test.go
+++ b/pkg/model/model_test.go
@@ -5,9 +5,21 @@ import (
"strings"
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
"github.com/stretchr/testify/require"
+=======
+<<<<<<< HEAD:v2/pkg/model/model_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/stretchr/testify/assert"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+ "github.com/stretchr/testify/require"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/model/model_test.go
+>>>>>>> projectdiscovery-main
"gopkg.in/yaml.v2"
)
diff --git a/pkg/model/types/severity/severities.go b/pkg/model/types/severity/severities.go
index e09ddca3ab..7ad55e1395 100644
--- a/pkg/model/types/severity/severities.go
+++ b/pkg/model/types/severity/severities.go
@@ -7,6 +7,14 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
"github.com/projectdiscovery/goflags"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/model/types/severity/severities.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/model/types/severity/severities.go
+>>>>>>> projectdiscovery-main
)
// Severities used by the goflags library for parsing an array of Severity types, passed as CLI arguments from the user
diff --git a/pkg/model/types/stringslice/stringslice.go b/pkg/model/types/stringslice/stringslice.go
index fb536fa14b..19d6644b80 100644
--- a/pkg/model/types/stringslice/stringslice.go
+++ b/pkg/model/types/stringslice/stringslice.go
@@ -5,8 +5,19 @@ import (
"fmt"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/utils"
"github.com/invopop/jsonschema"
+=======
+<<<<<<< HEAD:v2/pkg/model/types/stringslice/stringslice.go
+ "github.com/alecthomas/jsonschema"
+
+ "github.com/Explorer1092/nuclei/v2/pkg/utils"
+=======
+ "github.com/invopop/jsonschema"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/model/types/stringslice/stringslice.go
+>>>>>>> projectdiscovery-main
)
type StringOrSlice string
diff --git a/pkg/operators/common/dsl/dsl.go b/pkg/operators/common/dsl/dsl.go
index b8cbc39c7e..52d286be5e 100644
--- a/pkg/operators/common/dsl/dsl.go
+++ b/pkg/operators/common/dsl/dsl.go
@@ -10,6 +10,16 @@ import (
"github.com/miekg/dns"
"github.com/projectdiscovery/dsl"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/operators/common/dsl/dsl.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/dns/dnsclientpool"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/dns/dnsclientpool"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/operators/common/dsl/dsl.go
+>>>>>>> projectdiscovery-main
sliceutil "github.com/projectdiscovery/utils/slice"
stringsutil "github.com/projectdiscovery/utils/strings"
)
diff --git a/pkg/operators/extractors/compile.go b/pkg/operators/extractors/compile.go
index 03d73a406c..4569afda97 100644
--- a/pkg/operators/extractors/compile.go
+++ b/pkg/operators/extractors/compile.go
@@ -8,6 +8,14 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/operators/common/dsl"
"github.com/Knetic/govaluate"
"github.com/itchyny/gojq"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/operators/extractors/compile.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/common/dsl"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/operators/extractors/compile.go
+>>>>>>> projectdiscovery-main
)
// CompileExtractors performs the initial setup operation on an extractor
diff --git a/pkg/operators/extractors/extract.go b/pkg/operators/extractors/extract.go
index ddedde15bb..d77c200738 100644
--- a/pkg/operators/extractors/extract.go
+++ b/pkg/operators/extractors/extract.go
@@ -8,7 +8,15 @@ import (
"github.com/antchfx/htmlquery"
"github.com/antchfx/xmlquery"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/operators/extractors/extract.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/operators/extractors/extract.go
+>>>>>>> projectdiscovery-main
)
// ExtractRegex extracts text from a corpus and returns it
diff --git a/pkg/operators/matchers/compile.go b/pkg/operators/matchers/compile.go
index 994602819e..6b10870a33 100644
--- a/pkg/operators/matchers/compile.go
+++ b/pkg/operators/matchers/compile.go
@@ -8,7 +8,15 @@ import (
"github.com/Knetic/govaluate"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/operators/common/dsl"
+=======
+<<<<<<< HEAD:v2/pkg/operators/matchers/compile.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/common/dsl"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/operators/matchers/compile.go
+>>>>>>> projectdiscovery-main
)
// CompileMatchers performs the initial setup operation on a matcher
diff --git a/pkg/operators/matchers/match.go b/pkg/operators/matchers/match.go
index 078992e410..f54d264e72 100644
--- a/pkg/operators/matchers/match.go
+++ b/pkg/operators/matchers/match.go
@@ -12,12 +12,25 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/expressions"
dslRepo "github.com/projectdiscovery/dsl"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/operators/matchers/match.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/common/dsl"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/expressions"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+>>>>>>> projectdiscovery-main
stringsutil "github.com/projectdiscovery/utils/strings"
)
var (
// showDSLErr controls whether to show hidden DSL errors or not
showDSLErr = strings.EqualFold(os.Getenv("SHOW_DSL_ERRORS"), "true")
+<<<<<<< HEAD
+=======
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/operators/matchers/match.go
+>>>>>>> projectdiscovery-main
)
// MatchStatusCode matches a status code check against a corpus
diff --git a/pkg/operators/operators.go b/pkg/operators/operators.go
index 548c0c4c11..d4046d48e7 100644
--- a/pkg/operators/operators.go
+++ b/pkg/operators/operators.go
@@ -7,10 +7,24 @@ import (
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/operators/extractors"
"github.com/Explorer1092/nuclei/v3/pkg/operators/matchers"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/utils/excludematchers"
+=======
+<<<<<<< HEAD:v2/pkg/operators/operators.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/utils/excludematchers"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/excludematchers"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/operators/operators.go
+>>>>>>> projectdiscovery-main
sliceutil "github.com/projectdiscovery/utils/slice"
)
diff --git a/pkg/output/format_screen.go b/pkg/output/format_screen.go
index a7e5ef6b5f..21af14dd25 100644
--- a/pkg/output/format_screen.go
+++ b/pkg/output/format_screen.go
@@ -5,8 +5,17 @@ import (
"strconv"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
mapsutil "github.com/projectdiscovery/utils/maps"
+=======
+<<<<<<< HEAD:v2/pkg/output/format_screen.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ mapsutil "github.com/projectdiscovery/utils/maps"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/output/format_screen.go
+>>>>>>> projectdiscovery-main
)
// formatScreen formats the output for showing on screen.
@@ -39,6 +48,11 @@ func (w *StandardWriter) formatScreen(output *ResultEvent) []byte {
}
}
+ if output.GlobalMatchers {
+ builder.WriteString("] [")
+ builder.WriteString(w.aurora.BrightMagenta("global").String())
+ }
+
builder.WriteString("] [")
builder.WriteString(w.aurora.BrightBlue(output.Type).String())
builder.WriteString("] ")
diff --git a/pkg/output/output.go b/pkg/output/output.go
index 02df8ea145..c3cfa442ea 100644
--- a/pkg/output/output.go
+++ b/pkg/output/output.go
@@ -4,6 +4,8 @@ import (
"encoding/base64"
"fmt"
"io"
+ "log/slog"
+ "maps"
"os"
"path/filepath"
"regexp"
@@ -28,8 +30,32 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/interactsh/pkg/server"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/output/output.go
+ "github.com/Explorer1092/nuclei/v2/internal/colorizer"
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+ "github.com/Explorer1092/nuclei/v2/pkg/utils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/internal/colorizer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ protocolUtils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types/nucleierr"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+ "github.com/projectdiscovery/utils/errkit"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/output/output.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
osutils "github.com/projectdiscovery/utils/os"
+ unitutils "github.com/projectdiscovery/utils/unit"
+ urlutil "github.com/projectdiscovery/utils/url"
)
// Writer is an interface which writes output to somewhere for nuclei events.
@@ -66,6 +92,7 @@ type StandardWriter struct {
omitTemplate bool
DisableStdout bool
AddNewLinesOutputFile bool // by default this is only done for stdout
+ KeysToRedact []string
}
var decolorizerRegex = regexp.MustCompile(`\x1B\[[0-9;]*[a-zA-Z]`)
@@ -93,6 +120,15 @@ type InternalWrappedEvent struct {
InteractshMatched atomic.Bool
}
+func (iwe *InternalWrappedEvent) CloneShallow() *InternalWrappedEvent {
+ return &InternalWrappedEvent{
+ InternalEvent: maps.Clone(iwe.InternalEvent),
+ Results: nil,
+ OperatorsResult: nil,
+ UsesInteractsh: iwe.UsesInteractsh,
+ }
+}
+
func (iwe *InternalWrappedEvent) HasOperatorResult() bool {
iwe.RLock()
defer iwe.RUnlock()
@@ -168,6 +204,9 @@ type ResultEvent struct {
MatcherStatus bool `json:"matcher-status"`
// Lines is the line count for the specified match
Lines []int `json:"matched-line,omitempty"`
+ // GlobalMatchers identifies whether the matches was detected in the response
+ // of another template's result event
+ GlobalMatchers bool `json:"global-matchers,omitempty"`
// IssueTrackers is the metadata for issue trackers
IssueTrackers map[string]IssueTrackerMetadata `json:"issue_trackers,omitempty"`
@@ -182,8 +221,10 @@ type ResultEvent struct {
FuzzingMethod string `json:"fuzzing_method,omitempty"`
FuzzingParameter string `json:"fuzzing_parameter,omitempty"`
FuzzingPosition string `json:"fuzzing_position,omitempty"`
+ AnalyzerDetails string `json:"analyzer_details,omitempty"`
FileToIndexPosition map[string]int `json:"-"`
+ TemplateVerifier string `json:"-"`
Error string `json:"error,omitempty"`
}
@@ -248,6 +289,7 @@ func NewStandardWriter(options *types.Options) (*StandardWriter, error) {
storeResponse: options.StoreResponse,
storeResponseDir: options.StoreResponseDir,
omitTemplate: options.OmitTemplate,
+ KeysToRedact: options.Redact,
}
return writer, nil
}
@@ -256,7 +298,14 @@ func NewStandardWriter(options *types.Options) (*StandardWriter, error) {
func (w *StandardWriter) Write(event *ResultEvent) error {
// Enrich the result event with extra metadata on the template-path and url.
if event.TemplatePath != "" {
- event.Template, event.TemplateURL = utils.TemplatePathURL(types.ToString(event.TemplatePath), types.ToString(event.TemplateID))
+ event.Template, event.TemplateURL = utils.TemplatePathURL(types.ToString(event.TemplatePath), types.ToString(event.TemplateID), event.TemplateVerifier)
+ }
+
+ if len(w.KeysToRedact) > 0 {
+ event.Request = redactKeys(event.Request, w.KeysToRedact)
+ event.Response = redactKeys(event.Response, w.KeysToRedact)
+ event.CURLCommand = redactKeys(event.CURLCommand, w.KeysToRedact)
+ event.Matched = redactKeys(event.Matched, w.KeysToRedact)
}
event.Timestamp = time.Now()
@@ -297,12 +346,24 @@ func (w *StandardWriter) Write(event *ResultEvent) error {
return nil
}
+func redactKeys(data string, keysToRedact []string) string {
+ for _, key := range keysToRedact {
+ keyPattern := regexp.MustCompile(fmt.Sprintf(`(?i)(%s\s*[:=]\s*["']?)[^"'\r\n&]+(["'\r\n]?)`, regexp.QuoteMeta(key)))
+ data = keyPattern.ReplaceAllString(data, `$1***$2`)
+ }
+ return data
+}
+
// JSONLogRequest is a trace/error log request written to file
type JSONLogRequest struct {
- Template string `json:"template"`
- Input string `json:"input"`
- Error string `json:"error"`
- Type string `json:"type"`
+ Template string `json:"template"`
+ Type string `json:"type"`
+ Input string `json:"input"`
+ Timestamp *time.Time `json:"timestamp,omitempty"`
+ Address string `json:"address"`
+ Error string `json:"error"`
+ Kind string `json:"kind,omitempty"`
+ Attrs interface{} `json:"attrs,omitempty"`
}
// Request writes a log the requests trace log
@@ -315,12 +376,47 @@ func (w *StandardWriter) Request(templatePath, input, requestType string, reques
Input: input,
Type: requestType,
}
- if unwrappedErr := utils.UnwrapError(requestErr); unwrappedErr != nil {
- request.Error = unwrappedErr.Error()
- } else {
+ if w.timestamp {
+ ts := time.Now()
+ request.Timestamp = &ts
+ }
+ parsed, _ := urlutil.ParseAbsoluteURL(input, false)
+ if parsed != nil {
+ request.Address = parsed.Hostname()
+ port := parsed.Port()
+ if port == "" {
+ switch parsed.Scheme {
+ case urlutil.HTTP:
+ port = "80"
+ case urlutil.HTTPS:
+ port = "443"
+ }
+ }
+ request.Address += ":" + port
+ }
+ errX := errkit.FromError(requestErr)
+ if errX == nil {
request.Error = "none"
+ } else {
+ request.Kind = errkit.ErrKindUnknown.String()
+ var cause error
+ if len(errX.Errors()) > 1 {
+ cause = errX.Errors()[0]
+ }
+ if cause == nil {
+ cause = errX
+ }
+ cause = tryParseCause(cause)
+ request.Error = cause.Error()
+ request.Kind = errkit.GetErrorKind(requestErr, nucleierr.ErrTemplateLogic).String()
+ if len(errX.Attrs()) > 0 {
+ request.Attrs = slog.GroupValue(errX.Attrs()...)
+ }
+ }
+ // check if address slog attr is avaiable in error if set use it
+ if val := errkit.GetAttrValue(requestErr, "address"); val.Any() != nil {
+ request.Address = val.String()
}
-
data, err := jsoniter.Marshal(request)
if err != nil {
return
@@ -374,7 +470,7 @@ func (w *StandardWriter) WriteFailure(wrappedEvent *InternalWrappedEvent) error
// if no results were found, manually create a failure event
event := wrappedEvent.InternalEvent
- templatePath, templateURL := utils.TemplatePathURL(types.ToString(event["template-path"]), types.ToString(event["template-id"]))
+ templatePath, templateURL := utils.TemplatePathURL(types.ToString(event["template-path"]), types.ToString(event["template-id"]), types.ToString(event["template-verifier"]))
var templateInfo model.Info
if event["template-info"] != nil {
templateInfo = event["template-info"].(model.Info)
@@ -411,7 +507,7 @@ func (w *StandardWriter) WriteFailure(wrappedEvent *InternalWrappedEvent) error
return w.Write(data)
}
-var maxTemplateFileSizeForEncoding = 1024 * 1024
+var maxTemplateFileSizeForEncoding = unitutils.Mega
func (w *StandardWriter) encodeTemplate(templatePath string) string {
data, err := os.ReadFile(templatePath)
@@ -452,3 +548,24 @@ func (w *StandardWriter) WriteStoreDebugData(host, templateID, eventType string,
}
}
+
+// tryParseCause tries to parse the cause of given error
+// this is legacy support due to use of errorutil in existing libraries
+// but this should not be required once all libraries are updated
+func tryParseCause(err error) error {
+ if err == nil {
+ return nil
+ }
+ msg := err.Error()
+ if strings.HasPrefix(msg, "ReadStatusLine:") {
+ // last index is actual error (from rawhttp)
+ parts := strings.Split(msg, ":")
+ return errkit.New("%s", strings.TrimSpace(parts[len(parts)-1]))
+ }
+ if strings.Contains(msg, "read ") {
+ // same here
+ parts := strings.Split(msg, ":")
+ return errkit.New("%s", strings.TrimSpace(parts[len(parts)-1]))
+ }
+ return err
+}
diff --git a/pkg/output/output_test.go b/pkg/output/output_test.go
index af2b3a47b4..efd69a1f18 100644
--- a/pkg/output/output_test.go
+++ b/pkg/output/output_test.go
@@ -7,6 +7,14 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/pkg/errors"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/output/output_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/output/output_test.go
+>>>>>>> projectdiscovery-main
"github.com/stretchr/testify/require"
)
@@ -30,7 +38,7 @@ func TestStandardWriterRequest(t *testing.T) {
require.NoError(t, err)
w.Request("path", "input", "http", nil)
- require.Equal(t, `{"template":"path","input":"input","error":"none","type":"http"}`, traceWriter.String())
+ require.Equal(t, `{"template":"path","type":"http","input":"input","address":"input:","error":"none"}`, traceWriter.String())
require.Empty(t, errorWriter.String())
})
@@ -47,7 +55,7 @@ func TestStandardWriterRequest(t *testing.T) {
fmt.Errorf("GET https://example.com/tcpconfig.html/tcpconfig.html giving up after 2 attempts: %w", errors.New("context deadline exceeded (Client.Timeout exceeded while awaiting headers)")),
)
- require.Equal(t, `{"template":"misconfiguration/tcpconfig.yaml","input":"https://example.com/tcpconfig.html","error":"context deadline exceeded (Client.Timeout exceeded while awaiting headers)","type":"http"}`, errorWriter.String())
+ require.Equal(t, `{"template":"misconfiguration/tcpconfig.yaml","type":"http","input":"https://example.com/tcpconfig.html","address":"example.com:443","error":"context deadline exceeded (Client.Timeout exceeded while awaiting headers)","kind":"unknown-error"}`, errorWriter.String())
})
}
diff --git a/pkg/projectfile/httputil.go b/pkg/projectfile/httputil.go
index 3f00b8e811..dafeff3fd8 100644
--- a/pkg/projectfile/httputil.go
+++ b/pkg/projectfile/httputil.go
@@ -17,7 +17,7 @@ func hash(v interface{}) (string, error) {
sh := sha256.New()
- if _, err = io.WriteString(sh, string(data)); err != nil {
+ if _, err = sh.Write(data); err != nil {
return "", err
}
return hex.EncodeToString(sh.Sum(nil)), nil
diff --git a/pkg/protocols/code/code.go b/pkg/protocols/code/code.go
index ba56cad43e..55fb710545 100644
--- a/pkg/protocols/code/code.go
+++ b/pkg/protocols/code/code.go
@@ -32,16 +32,19 @@ import (
"github.com/projectdiscovery/gozero"
gozerotypes "github.com/projectdiscovery/gozero/types"
contextutil "github.com/projectdiscovery/utils/context"
+ "github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
)
const (
- pythonEnvRegex = `os\.getenv\(['"]([^'"]+)['"]\)`
- TimeoutMultiplier = 6 // timeout multiplier for code protocol
+ pythonEnvRegex = `os\.getenv\(['"]([^'"]+)['"]\)`
)
var (
+ // pythonEnvRegexCompiled is the compiled regex for python environment variables
pythonEnvRegexCompiled = regexp.MustCompile(pythonEnvRegex)
+ // ErrCodeExecutionDeadline is the error returned when alloted time for script execution exceeds
+ ErrCodeExecutionDeadline = errkit.New("code execution deadline exceeded").SetKind(errkit.ErrKindDeadline).Build()
)
// Request is a request for the SSL protocol
@@ -83,6 +86,12 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
Args: request.Args,
EarlyCloseFileDescriptor: true,
}
+
+ if options.Options.Debug || options.Options.DebugResponse {
+ // enable debug mode for gozero
+ gozeroOptions.DebugMode = true
+ }
+
engine, err := gozero.New(gozeroOptions)
if err != nil {
return errorutil.NewWithErr(err).Msgf("[%s] engines '%s' not available on host", options.TemplateID, strings.Join(request.Engine, ","))
@@ -175,9 +184,6 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
metaSrc.AddVariable(gozerotypes.Variable{Name: name, Value: v})
}
- // set timeout using multiplier
- timeout := TimeoutMultiplier * request.options.Options.Timeout
-
if request.PreCondition != "" {
if request.options.Options.Debug || request.options.Options.DebugRequests {
gologger.Debug().Msgf("[%s] Executing Precondition for Code request\n", request.TemplateID)
@@ -195,11 +201,11 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
result, err := request.options.JsCompiler.ExecuteWithOptions(request.preConditionCompiled, args,
&compiler.ExecuteOptions{
- Timeout: timeout,
- Source: &request.PreCondition,
- Callback: registerPreConditionFunctions,
- Cleanup: cleanUpPreConditionFunctions,
- Context: input.Context(),
+ TimeoutVariants: request.options.Options.GetTimeouts(),
+ Source: &request.PreCondition,
+ Callback: registerPreConditionFunctions,
+ Cleanup: cleanUpPreConditionFunctions,
+ Context: input.Context(),
})
if err != nil {
return errorutil.NewWithTag(request.TemplateID, "could not execute pre-condition: %s", err)
@@ -214,7 +220,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
}
}
- ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeout)*time.Second)
+ ctx, cancel := context.WithTimeoutCause(input.Context(), request.options.Options.GetTimeouts().CodeExecutionTimeout, ErrCodeExecutionDeadline)
defer cancel()
// Note: we use contextutil despite the fact that gozero accepts context as argument
gOutput, err := contextutil.ExecFuncWithTwoReturns(ctx, func() (*gozerotypes.Result, error) {
@@ -235,11 +241,26 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
gologger.Verbose().Msgf("[%s] Executed code on local machine %v", request.options.TemplateID, input.MetaInput.Input)
if vardump.EnableVarDump {
- gologger.Debug().Msgf("Code Protocol request variables: \n%s\n", vardump.DumpVariables(allvars))
+ gologger.Debug().Msgf("Code Protocol request variables: %s\n", vardump.DumpVariables(allvars))
}
if request.options.Options.Debug || request.options.Options.DebugRequests {
- gologger.Debug().Msgf("[%s] Dumped Executed Source Code for %v\n\n%v\n", request.options.TemplateID, input.MetaInput.Input, interpretEnvVars(request.Source, allvars))
+ gologger.Debug().MsgFunc(func() string {
+ dashes := strings.Repeat("-", 15)
+ sb := &strings.Builder{}
+ sb.WriteString(fmt.Sprintf("[%s] Dumped Executed Source Code for input/stdin: '%v'", request.options.TemplateID, input.MetaInput.Input))
+ sb.WriteString(fmt.Sprintf("\n%v\n%v\n%v\n", dashes, "Source Code:", dashes))
+ sb.WriteString(interpretEnvVars(request.Source, allvars))
+ sb.WriteString("\n")
+ sb.WriteString(fmt.Sprintf("\n%v\n%v\n%v\n", dashes, "Command Executed:", dashes))
+ sb.WriteString(interpretEnvVars(gOutput.Command, allvars))
+ sb.WriteString("\n")
+ sb.WriteString(fmt.Sprintf("\n%v\n%v\n%v\n", dashes, "Command Output:", dashes))
+ sb.WriteString(gOutput.DebugData.String())
+ sb.WriteString("\n")
+ sb.WriteString("[WRN] Command Output here is stdout+sterr, in response variables they are seperate (use -v -svd flags for more details)")
+ return sb.String()
+ })
}
dataOutputString := fmtStdout(gOutput.Stdout.String())
@@ -348,6 +369,7 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(request.options.TemplateID),
TemplatePath: types.ToString(request.options.TemplatePath),
Info: request.options.TemplateInfo,
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Matched: types.ToString(wrapped.InternalEvent["input"]),
Host: fields.Host,
diff --git a/pkg/protocols/common/automaticscan/automaticscan.go b/pkg/protocols/common/automaticscan/automaticscan.go
index fc2c7bcc28..a072ecccd2 100644
--- a/pkg/protocols/common/automaticscan/automaticscan.go
+++ b/pkg/protocols/common/automaticscan/automaticscan.go
@@ -32,13 +32,14 @@ import (
sliceutil "github.com/projectdiscovery/utils/slice"
stringsutil "github.com/projectdiscovery/utils/strings"
syncutil "github.com/projectdiscovery/utils/sync"
+ unitutils "github.com/projectdiscovery/utils/unit"
wappalyzer "github.com/projectdiscovery/wappalyzergo"
"gopkg.in/yaml.v2"
)
const (
mappingFilename = "wappalyzer-mapping.yml"
- maxDefaultBody = 4 * 1024 * 1024 // 4MB
+ maxDefaultBody = 4 * unitutils.Mega
)
// Options contains configuration options for automatic scan service
diff --git a/pkg/protocols/common/contextargs/contextargs.go b/pkg/protocols/common/contextargs/contextargs.go
index 8e8f0361c4..af55595498 100644
--- a/pkg/protocols/common/contextargs/contextargs.go
+++ b/pkg/protocols/common/contextargs/contextargs.go
@@ -37,15 +37,24 @@ func New(ctx context.Context) *Context {
return NewWithInput(ctx, "")
}
+// NewWithMetaInput creates a new contextargs instance with meta input
+func NewWithMetaInput(ctx context.Context, input *MetaInput) *Context {
+ n := New(ctx)
+ n.MetaInput = input
+ return n
+}
+
// Create a new contextargs instance with input string
func NewWithInput(ctx context.Context, input string) *Context {
jar, err := cookiejar.New(nil)
if err != nil {
gologger.Error().Msgf("contextargs: could not create cookie jar: %s\n", err)
}
+ metaInput := NewMetaInput()
+ metaInput.Input = input
return &Context{
ctx: ctx,
- MetaInput: &MetaInput{Input: input},
+ MetaInput: metaInput,
CookieJar: jar,
args: &mapsutil.SyncLockMap[string, interface{}]{
Map: make(map[string]interface{}),
@@ -174,3 +183,20 @@ func (ctx *Context) Clone() *Context {
}
return newCtx
}
+
+// GetCopyIfHostOutdated returns a new contextargs if the host is outdated
+func GetCopyIfHostOutdated(ctx *Context, url string) *Context {
+ if ctx.MetaInput.Input == "" {
+ newctx := ctx.Clone()
+ newctx.MetaInput.Input = url
+ return newctx
+ }
+ orig, _ := urlutil.Parse(ctx.MetaInput.Input)
+ newURL, _ := urlutil.Parse(url)
+ if orig != nil && newURL != nil && orig.Host != newURL.Host {
+ newCtx := ctx.Clone()
+ newCtx.MetaInput.Input = newURL.Host
+ return newCtx
+ }
+ return ctx
+}
diff --git a/pkg/protocols/common/contextargs/metainput.go b/pkg/protocols/common/contextargs/metainput.go
index 7fe2ad580b..6755caad03 100644
--- a/pkg/protocols/common/contextargs/metainput.go
+++ b/pkg/protocols/common/contextargs/metainput.go
@@ -4,11 +4,14 @@ import (
"bytes"
"crypto/md5"
"fmt"
+ "net"
"strings"
+ "sync"
"github.com/Explorer1092/nuclei/v3/pkg/input/types"
jsoniter "github.com/json-iterator/go"
urlutil "github.com/projectdiscovery/utils/url"
+ "github.com/segmentio/ksuid"
)
// MetaInput represents a target with metadata (TODO: replace with https://github.com/projectdiscovery/metainput)
@@ -22,6 +25,12 @@ type MetaInput struct {
// ReqResp is the raw request for the input
ReqResp *types.RequestResponse `json:"raw-request,omitempty"`
+
+ mu *sync.Mutex
+}
+
+func NewMetaInput() *MetaInput {
+ return &MetaInput{mu: &sync.Mutex{}}
}
func (metaInput *MetaInput) marshalToBuffer() (bytes.Buffer, error) {
@@ -47,6 +56,56 @@ func (metaInput *MetaInput) URL() (*urlutil.URL, error) {
return instance, nil
}
+// Port returns the port of the target
+// if port is not present then empty string is returned
+func (metaInput *MetaInput) Port() string {
+ target, err := urlutil.ParseAbsoluteURL(metaInput.Input, false)
+ if err != nil {
+ return ""
+ }
+ return target.Port()
+}
+
+// Address return the remote address of target
+// Note: it does not resolve the domain to ip
+// it is meant to be used by hosterrorsCache if invalid metainput
+// is provided then it uses a random ksuid as hostname to avoid skipping valid targets
+func (metaInput *MetaInput) Address() string {
+ var hostname, port string
+ target, err := urlutil.ParseAbsoluteURL(metaInput.Target(), false)
+ if err != nil {
+ if metaInput.CustomIP == "" {
+ // since this is used in hosterrorscache we add a random id
+ // which will never be used to avoid skipping valid targets
+ hostname = fmt.Sprintf("invalid-%s", ksuid.New().String())
+ }
+ } else {
+ hostname = target.Hostname()
+ port = target.Port()
+ if port == "" {
+ switch target.Scheme {
+ case urlutil.HTTP:
+ port = "80"
+ case urlutil.HTTPS:
+ port = "443"
+ default:
+ port = "80"
+ }
+ }
+ }
+ if metaInput.CustomIP != "" {
+ hostname = metaInput.CustomIP
+ }
+ if port == "" {
+ if strings.HasPrefix(hostname, "http://") {
+ port = "80"
+ } else if strings.HasPrefix(hostname, "https://") {
+ port = "443"
+ }
+ }
+ return net.JoinHostPort(hostname, port)
+}
+
// ID returns a unique id/hash for metainput
func (metaInput *MetaInput) ID() string {
if metaInput.CustomIP != "" {
@@ -83,10 +142,13 @@ func (metaInput *MetaInput) Unmarshal(data string) error {
}
func (metaInput *MetaInput) Clone() *MetaInput {
- input := &MetaInput{
- Input: metaInput.Input,
- CustomIP: metaInput.CustomIP,
- }
+ metaInput.mu.Lock()
+ defer metaInput.mu.Unlock()
+
+ input := NewMetaInput()
+ input.Input = metaInput.Input
+ input.CustomIP = metaInput.CustomIP
+ input.hash = metaInput.hash
if metaInput.ReqResp != nil {
input.ReqResp = metaInput.ReqResp.Clone()
}
@@ -108,6 +170,9 @@ func (metaInput *MetaInput) GetScanHash(templateId string) string {
// there may be some cases where metainput is changed ex: while executing self-contained template etc
// but that totally changes the scanID/hash so to avoid that we compute hash only once
// and reuse it for all subsequent calls
+ metaInput.mu.Lock()
+ defer metaInput.mu.Unlock()
+
if metaInput.hash == "" {
var rawRequest string
if metaInput.ReqResp != nil {
diff --git a/pkg/protocols/common/expressions/expressions.go b/pkg/protocols/common/expressions/expressions.go
index b2338856c0..abb7389013 100644
--- a/pkg/protocols/common/expressions/expressions.go
+++ b/pkg/protocols/common/expressions/expressions.go
@@ -5,9 +5,21 @@ import (
"github.com/Knetic/govaluate"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/operators/common/dsl"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/marker"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/replacer"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/expressions/expressions.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/common/dsl"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/marker"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/replacer"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/marker"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/replacer"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/expressions/expressions.go
+>>>>>>> projectdiscovery-main
stringsutil "github.com/projectdiscovery/utils/strings"
)
diff --git a/pkg/protocols/common/expressions/variables.go b/pkg/protocols/common/expressions/variables.go
index 46684691a8..79f82ee304 100644
--- a/pkg/protocols/common/expressions/variables.go
+++ b/pkg/protocols/common/expressions/variables.go
@@ -7,6 +7,14 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/operators/common/dsl"
"github.com/Knetic/govaluate"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/expressions/variables.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/common/dsl"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/expressions/variables.go
+>>>>>>> projectdiscovery-main
)
var (
diff --git a/pkg/protocols/common/generators/generators.go b/pkg/protocols/common/generators/generators.go
index 2abd34ff50..24bb9b7499 100644
--- a/pkg/protocols/common/generators/generators.go
+++ b/pkg/protocols/common/generators/generators.go
@@ -5,8 +5,18 @@ package generators
import (
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/generators/generators.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/generators/generators.go
+>>>>>>> projectdiscovery-main
)
// PayloadGenerator is the generator struct for generating payloads
@@ -25,8 +35,19 @@ func New(payloads map[string]interface{}, attackType AttackType, templatePath st
// Resolve payload paths if they are files.
payloadsFinal := make(map[string]interface{})
- for name, payload := range payloads {
- payloadsFinal[name] = payload
+ for payloadName, v := range payloads {
+ switch value := v.(type) {
+ case map[interface{}]interface{}:
+ values, err := parsePayloadsWithAggression(payloadName, value, opts.FuzzAggressionLevel)
+ if err != nil {
+ return nil, errors.Wrap(err, "could not parse payloads with aggression")
+ }
+ for k, v := range values {
+ payloadsFinal[k] = v
+ }
+ default:
+ payloadsFinal[payloadName] = v
+ }
}
generator := &PayloadGenerator{catalog: catalog, options: opts}
@@ -57,6 +78,60 @@ func New(payloads map[string]interface{}, attackType AttackType, templatePath st
return generator, nil
}
+type aggressionLevelToPayloads struct {
+ Low []interface{}
+ Medium []interface{}
+ High []interface{}
+}
+
+// parsePayloadsWithAggression parses the payloads with the aggression level
+//
+// Three agression are supported -
+// - low
+// - medium
+// - high
+//
+// low is the default level. If medium is specified, all templates from
+// low and medium are executed. Similarly with high, including all templates
+// from low, medium, high.
+func parsePayloadsWithAggression(name string, v map[interface{}]interface{}, agression string) (map[string]interface{}, error) {
+ payloadsLevels := &aggressionLevelToPayloads{}
+
+ for k, v := range v {
+ if _, ok := v.([]interface{}); !ok {
+ return nil, errors.Errorf("only lists are supported for aggression levels payloads")
+ }
+ var ok bool
+ switch k {
+ case "low":
+ payloadsLevels.Low, ok = v.([]interface{})
+ case "medium":
+ payloadsLevels.Medium, ok = v.([]interface{})
+ case "high":
+ payloadsLevels.High, ok = v.([]interface{})
+ default:
+ return nil, errors.Errorf("invalid aggression level %s specified for %s", k, name)
+ }
+ if !ok {
+ return nil, errors.Errorf("invalid aggression level %s specified for %s", k, name)
+ }
+ }
+
+ payloads := make(map[string]interface{})
+ switch agression {
+ case "low":
+ payloads[name] = payloadsLevels.Low
+ case "medium":
+ payloads[name] = append(payloadsLevels.Low, payloadsLevels.Medium...)
+ case "high":
+ payloads[name] = append(payloadsLevels.Low, payloadsLevels.Medium...)
+ payloads[name] = append(payloads[name].([]interface{}), payloadsLevels.High...)
+ default:
+ return nil, errors.Errorf("invalid aggression level %s specified for %s", agression, name)
+ }
+ return payloads, nil
+}
+
// Iterator is a single instance of an iterator for a generator structure
type Iterator struct {
Type AttackType
diff --git a/pkg/protocols/common/generators/generators_test.go b/pkg/protocols/common/generators/generators_test.go
index 18ae844f77..6b111fcb19 100644
--- a/pkg/protocols/common/generators/generators_test.go
+++ b/pkg/protocols/common/generators/generators_test.go
@@ -1,12 +1,23 @@
package generators
import (
+ "strings"
"testing"
"github.com/stretchr/testify/require"
+ "gopkg.in/yaml.v2"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/disk"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/generators/generators_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/disk"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/generators/generators_test.go
+>>>>>>> projectdiscovery-main
)
func TestBatteringRamGenerator(t *testing.T) {
@@ -90,3 +101,49 @@ func getOptions(allowLocalFileAccess bool) *types.Options {
opts.AllowLocalFileAccess = allowLocalFileAccess
return opts
}
+
+func TestParsePayloadsWithAggression(t *testing.T) {
+ testPayload := `linux_path:
+ low:
+ - /etc/passwd
+ medium:
+ - ../etc/passwd
+ - ../../etc/passwd
+ high:
+ - ../../../etc/passwd
+ - ../../../../etc/passwd
+ - ../../../../../etc/passwd`
+
+ var payloads map[string]interface{}
+ err := yaml.NewDecoder(strings.NewReader(testPayload)).Decode(&payloads)
+ require.Nil(t, err, "could not unmarshal yaml")
+
+ aggressionsToValues := map[string][]string{
+ "low": {
+ "/etc/passwd",
+ },
+ "medium": {
+ "/etc/passwd",
+ "../etc/passwd",
+ "../../etc/passwd",
+ },
+ "high": {
+ "/etc/passwd",
+ "../etc/passwd",
+ "../../etc/passwd",
+ "../../../etc/passwd",
+ "../../../../etc/passwd",
+ "../../../../../etc/passwd",
+ },
+ }
+
+ for k, v := range payloads {
+ for aggression, values := range aggressionsToValues {
+ parsed, err := parsePayloadsWithAggression(k, v.(map[interface{}]interface{}), aggression)
+ require.Nil(t, err, "could not parse payloads with aggression")
+
+ gotValues := parsed[k].([]interface{})
+ require.Equal(t, len(values), len(gotValues), "could not get correct number of values")
+ }
+ }
+}
diff --git a/pkg/protocols/common/generators/options.go b/pkg/protocols/common/generators/options.go
index 4d59b0c4cb..c74060de7e 100644
--- a/pkg/protocols/common/generators/options.go
+++ b/pkg/protocols/common/generators/options.go
@@ -1,7 +1,15 @@
package generators
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/generators/options.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/generators/options.go
+>>>>>>> projectdiscovery-main
)
// BuildPayloadFromOptions returns a map with the payloads provided via CLI
diff --git a/pkg/protocols/common/generators/validate.go b/pkg/protocols/common/generators/validate.go
index 6677b7283e..2bd0c2d96f 100644
--- a/pkg/protocols/common/generators/validate.go
+++ b/pkg/protocols/common/generators/validate.go
@@ -6,8 +6,17 @@ import (
"path/filepath"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/generators/validate.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/generators/validate.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
folderutil "github.com/projectdiscovery/utils/folder"
)
@@ -22,6 +31,13 @@ func (g *PayloadGenerator) validate(payloads map[string]interface{}, templatePat
return errors.New("invalid number of lines in payload")
}
+ // For historical reasons, "validate" checks to see if the payload file exist.
+ // If we're using a custom helper function, then we need to skip any validation beyond just checking the string syntax.
+ // Actually attempting to load the file will determine whether or not it exists.
+ if g.options.LoadHelperFileFunction != nil {
+ return nil
+ }
+
// check if it's a file and try to load it
if fileutil.FileExists(payloadType) {
continue
diff --git a/pkg/protocols/common/globalmatchers/globalmatchers.go b/pkg/protocols/common/globalmatchers/globalmatchers.go
new file mode 100644
index 0000000000..321759f927
--- /dev/null
+++ b/pkg/protocols/common/globalmatchers/globalmatchers.go
@@ -0,0 +1,84 @@
+package globalmatchers
+
+import (
+ "maps"
+
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+)
+
+// Storage is a struct that holds the global matchers
+type Storage struct {
+ requests []*Item
+}
+
+// Callback is called when a global matcher is matched.
+// It receives internal event & result of the operator execution.
+type Callback func(event output.InternalEvent, result *operators.Result)
+
+// Item is a struct that holds the global matchers
+// details for a template
+type Item struct {
+ TemplateID string
+ TemplatePath string
+ TemplateInfo model.Info
+ Operators []*operators.Operators
+}
+
+// New creates a new storage for global matchers
+func New() *Storage {
+ return &Storage{requests: make([]*Item, 0)}
+}
+
+// hasStorage checks if the Storage is initialized
+func (s *Storage) hasStorage() bool {
+ return s != nil
+}
+
+// AddOperator adds a new operator to the global matchers
+func (s *Storage) AddOperator(item *Item) {
+ if !s.hasStorage() {
+ return
+ }
+
+ s.requests = append(s.requests, item)
+}
+
+// HasMatchers returns true if we have global matchers
+func (s *Storage) HasMatchers() bool {
+ if !s.hasStorage() {
+ return false
+ }
+
+ return len(s.requests) > 0
+}
+
+// Match matches the global matchers against the response
+func (s *Storage) Match(
+ event output.InternalEvent,
+ matchFunc operators.MatchFunc,
+ extractFunc operators.ExtractFunc,
+ isDebug bool,
+ callback Callback,
+) {
+ for _, item := range s.requests {
+ for _, operator := range item.Operators {
+ newEvent := maps.Clone(event)
+ newEvent.Set("origin-template-id", event["template-id"])
+ newEvent.Set("origin-template-info", event["template-info"])
+ newEvent.Set("origin-template-path", event["template-path"])
+ newEvent.Set("template-id", item.TemplateID)
+ newEvent.Set("template-info", item.TemplateInfo)
+ newEvent.Set("template-path", item.TemplatePath)
+ newEvent.Set("global-matchers", true)
+
+ result, matched := operator.Execute(newEvent, matchFunc, extractFunc, isDebug)
+ if !matched {
+ continue
+ }
+
+ callback(newEvent, result)
+ }
+ }
+}
diff --git a/pkg/protocols/common/helpers/eventcreator/eventcreator.go b/pkg/protocols/common/helpers/eventcreator/eventcreator.go
index 4a848eb4a2..bef99e5629 100644
--- a/pkg/protocols/common/helpers/eventcreator/eventcreator.go
+++ b/pkg/protocols/common/helpers/eventcreator/eventcreator.go
@@ -6,8 +6,24 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/utils/vardump"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
"golang.org/x/text/cases"
"golang.org/x/text/language"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/helpers/eventcreator/eventcreator.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/utils/vardump"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ "golang.org/x/text/cases"
+ "golang.org/x/text/language"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/helpers/eventcreator/eventcreator.go
+>>>>>>> projectdiscovery-main
)
// CreateEvent wraps the outputEvent with the result of the operators defined on the request
@@ -24,7 +40,7 @@ func CreateEventWithAdditionalOptions(request protocols.Request, outputEvent out
// Dump response variables if ran in debug mode
if vardump.EnableVarDump {
protoName := cases.Title(language.English).String(request.Type().String())
- gologger.Debug().Msgf("%v Protocol response variables: \n%s\n", protoName, vardump.DumpVariables(outputEvent))
+ gologger.Debug().Msgf("%v Protocol response variables: %s\n", protoName, vardump.DumpVariables(outputEvent))
}
for _, compiledOperator := range request.GetCompiledOperators() {
if compiledOperator != nil {
diff --git a/pkg/protocols/common/helpers/responsehighlighter/hexdump.go b/pkg/protocols/common/helpers/responsehighlighter/hexdump.go
index decc985c72..36d60c34da 100644
--- a/pkg/protocols/common/helpers/responsehighlighter/hexdump.go
+++ b/pkg/protocols/common/helpers/responsehighlighter/hexdump.go
@@ -44,7 +44,8 @@ func toHighLightedHexDump(hexDump, snippetToHighlight string) (HighlightableHexD
hexDumpRowValues := hexDumpParsePattern.FindAllStringSubmatch(hexDump, -1)
if hexDumpRowValues == nil || len(hexDumpRowValues) != strings.Count(hexDump, "\n") {
message := "could not parse hexdump"
- gologger.Warning().Msgf(message)
+ gologger.Warning().Msg(message)
+
return HighlightableHexDump{}, errors.New(message)
}
diff --git a/pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go b/pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go
index 5325e4aac6..ead3d17e7c 100644
--- a/pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go
+++ b/pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go
@@ -7,7 +7,15 @@ import (
"github.com/logrusorgru/aurora"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/operators"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go
+>>>>>>> projectdiscovery-main
)
var colorFunction = aurora.Green
diff --git a/pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go b/pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go
index 815cfb01de..2eaa823983 100644
--- a/pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go
+++ b/pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go
@@ -4,8 +4,19 @@ import (
"encoding/hex"
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/stretchr/testify/require"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go
+ "github.com/stretchr/testify/assert"
+
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/stretchr/testify/require"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go
+>>>>>>> projectdiscovery-main
)
const input = "abcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmn"
diff --git a/pkg/protocols/common/helpers/writer/writer.go b/pkg/protocols/common/helpers/writer/writer.go
index cf00e98111..99fba3523d 100644
--- a/pkg/protocols/common/helpers/writer/writer.go
+++ b/pkg/protocols/common/helpers/writer/writer.go
@@ -5,6 +5,18 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/progress"
"github.com/Explorer1092/nuclei/v3/pkg/reporting"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/helpers/writer/writer.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/progress"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/progress"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/helpers/writer/writer.go
+>>>>>>> projectdiscovery-main
)
// WriteResult is a helper for writing results to the output
diff --git a/pkg/protocols/common/hosterrorscache/hosterrorscache.go b/pkg/protocols/common/hosterrorscache/hosterrorscache.go
index 3abd1ec55f..bca4803e8a 100644
--- a/pkg/protocols/common/hosterrorscache/hosterrorscache.go
+++ b/pkg/protocols/common/hosterrorscache/hosterrorscache.go
@@ -8,19 +8,28 @@ import (
"sync"
"sync/atomic"
- "github.com/bluele/gcache"
+ "github.com/Mzack9999/gcache"
"github.com/projectdiscovery/gologger"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types/nucleierr"
+ "github.com/projectdiscovery/utils/errkit"
+ stringsutil "github.com/projectdiscovery/utils/strings"
)
// CacheInterface defines the signature of the hosterrorscache so that
// users of Nuclei as embedded lib may implement their own cache
type CacheInterface interface {
- SetVerbose(verbose bool) // log verbosely
- Close() // close the cache
- Check(value string) bool // return true if the host should be skipped
- MarkFailed(value string, err error) // record a failure (and cause) for the host
+ SetVerbose(verbose bool) // log verbosely
+ Close() // close the cache
+ Check(protoType string, ctx *contextargs.Context) bool // return true if the host should be skipped
+ MarkFailed(protoType string, ctx *contextargs.Context, err error) // record a failure (and cause) for the host
}
+var (
+ _ CacheInterface = (*Cache)(nil)
+)
+
// Cache is a cache for host based errors. It allows skipping
// certain hosts based on an error threshold.
//
@@ -29,20 +38,22 @@ type CacheInterface interface {
type Cache struct {
MaxHostError int
verbose bool
- failedTargets gcache.Cache
+ failedTargets gcache.Cache[string, *cacheItem]
TrackError []string
}
type cacheItem struct {
- errors atomic.Int32
sync.Once
+ errors atomic.Int32
+ isPermanentErr bool
+ cause error // optional cause
}
const DefaultMaxHostsCount = 10000
// New returns a new host max errors cache
func New(maxHostError, maxHostsCount int, trackError []string) *Cache {
- gc := gcache.New(maxHostsCount).
+ gc := gcache.New[string, *cacheItem](maxHostsCount).
ARC().
Build()
return &Cache{failedTargets: gc, MaxHostError: maxHostError, TrackError: trackError}
@@ -55,27 +66,43 @@ func (c *Cache) SetVerbose(verbose bool) {
// Close closes the host errors cache
func (c *Cache) Close() {
+ if config.DefaultConfig.IsDebugArgEnabled(config.DebugArgHostErrorStats) {
+ items := c.failedTargets.GetALL(false)
+ for k, val := range items {
+ gologger.Info().Label("MaxHostErrorStats").Msgf("Host: %s, Errors: %d", k, val.errors.Load())
+ }
+ }
c.failedTargets.Purge()
}
-func (c *Cache) normalizeCacheValue(value string) string {
- finalValue := value
- if strings.HasPrefix(value, "http") {
- if parsed, err := url.Parse(value); err == nil {
- hostname := parsed.Host
- finalPort := parsed.Port()
- if finalPort == "" {
- if parsed.Scheme == "https" {
- finalPort = "443"
- } else {
- finalPort = "80"
- }
- hostname = net.JoinHostPort(parsed.Host, finalPort)
+// NormalizeCacheValue processes the input value and returns a normalized cache
+// value.
+func (c *Cache) NormalizeCacheValue(value string) string {
+ var normalizedValue string = value
+
+ u, err := url.ParseRequestURI(value)
+ if err != nil || u.Host == "" {
+ u, err2 := url.ParseRequestURI("https://" + value)
+ if err2 != nil {
+ return normalizedValue
+ }
+
+ normalizedValue = u.Host
+ } else {
+ port := u.Port()
+ if port == "" {
+ switch u.Scheme {
+ case "https":
+ normalizedValue = net.JoinHostPort(u.Host, "443")
+ case "http":
+ normalizedValue = net.JoinHostPort(u.Host, "80")
}
- finalValue = hostname
+ } else {
+ normalizedValue = u.Host
}
}
- return finalValue
+
+ return normalizedValue
}
// ErrUnresponsiveHost is returned when a host is unresponsive
@@ -88,18 +115,22 @@ func (c *Cache) normalizeCacheValue(value string) string {
// - URL: https?:// type
// - Host:port type
// - host type
-func (c *Cache) Check(value string) bool {
- finalValue := c.normalizeCacheValue(value)
+func (c *Cache) Check(protoType string, ctx *contextargs.Context) bool {
+ finalValue := c.GetKeyFromContext(ctx, nil)
existingCacheItem, err := c.failedTargets.GetIFPresent(finalValue)
if err != nil {
return false
}
- existingCacheItemValue := existingCacheItem.(*cacheItem)
+ if existingCacheItem.isPermanentErr {
+ // skipping permanent errors is expected so verbose instead of info
+ gologger.Verbose().Msgf("Skipped %s from target list as found unresponsive permanently: %s", finalValue, existingCacheItem.cause)
+ return true
+ }
- if existingCacheItemValue.errors.Load() >= int32(c.MaxHostError) {
- existingCacheItemValue.Do(func() {
- gologger.Info().Msgf("Skipped %s from target list as found unresponsive %d times", finalValue, existingCacheItemValue.errors.Load())
+ if existingCacheItem.errors.Load() >= int32(c.MaxHostError) {
+ existingCacheItem.Do(func() {
+ gologger.Info().Msgf("Skipped %s from target list as found unresponsive %d times", finalValue, existingCacheItem.errors.Load())
})
return true
}
@@ -107,36 +138,103 @@ func (c *Cache) Check(value string) bool {
}
// MarkFailed marks a host as failed previously
-func (c *Cache) MarkFailed(value string, err error) {
- if !c.checkError(err) {
+func (c *Cache) MarkFailed(protoType string, ctx *contextargs.Context, err error) {
+ if !c.checkError(protoType, err) {
return
}
- finalValue := c.normalizeCacheValue(value)
+ finalValue := c.GetKeyFromContext(ctx, err)
existingCacheItem, err := c.failedTargets.GetIFPresent(finalValue)
if err != nil || existingCacheItem == nil {
newItem := &cacheItem{errors: atomic.Int32{}}
newItem.errors.Store(1)
+ if errkit.IsKind(err, errkit.ErrKindNetworkPermanent) {
+ // skip this address altogether
+ // permanent errors are always permanent hence this is created once
+ // and never updated so no need to synchronize
+ newItem.isPermanentErr = true
+ newItem.cause = err
+ }
_ = c.failedTargets.Set(finalValue, newItem)
return
}
- existingCacheItemValue := existingCacheItem.(*cacheItem)
- existingCacheItemValue.errors.Add(1)
- _ = c.failedTargets.Set(finalValue, existingCacheItemValue)
+ existingCacheItem.errors.Add(1)
+ _ = c.failedTargets.Set(finalValue, existingCacheItem)
}
-var reCheckError = regexp.MustCompile(`(no address found for host|Client\.Timeout exceeded while awaiting headers|could not resolve host|connection refused|connection reset by peer|i/o timeout|could not connect to any address found for host|timeout awaiting response headers)`)
+// GetKeyFromContext returns the key for the cache from the context
+func (c *Cache) GetKeyFromContext(ctx *contextargs.Context, err error) string {
+ // Note:
+ // ideally any changes made to remote addr in template like {{Hostname}}:81 etc
+ // should be reflected in contextargs but it is not yet reflected in some cases
+ // and needs refactor of ScanContext + ContextArgs to achieve that
+ // i.e why we use real address from error if present
+ address := ctx.MetaInput.Address()
+ // get address override from error
+ if err != nil {
+ tmp := errkit.GetAttrValue(err, "address")
+ if tmp.Any() != nil {
+ address = tmp.String()
+ }
+ }
+ finalValue := c.NormalizeCacheValue(address)
+ return finalValue
+}
+
+var reCheckError = regexp.MustCompile(`(no address found for host|could not resolve host|connection refused|connection reset by peer|could not connect to any address found for host|timeout awaiting response headers)`)
// checkError checks if an error represents a type that should be
// added to the host skipping table.
-func (c *Cache) checkError(err error) bool {
+// it first parses error and extracts the cause and checks for blacklisted
+// or common errors that should be skipped
+func (c *Cache) checkError(protoType string, err error) bool {
if err == nil {
return false
}
- errString := err.Error()
- for _, msg := range c.TrackError {
- if strings.Contains(errString, msg) {
- return true
+ if protoType != "http" {
+ return false
+ }
+ kind := errkit.GetErrorKind(err, nucleierr.ErrTemplateLogic)
+ switch kind {
+ case nucleierr.ErrTemplateLogic:
+ // these are errors that are not related to the target
+ // and are due to template logic
+ return false
+ case errkit.ErrKindNetworkTemporary:
+ // these should not be counted as host errors
+ return false
+ case errkit.ErrKindNetworkPermanent:
+ // these should be counted as host errors
+ return true
+ case errkit.ErrKindDeadline:
+ // these should not be counted as host errors
+ return false
+ default:
+ // parse error for further processing
+ errX := errkit.FromError(err)
+ tmp := errX.Cause()
+ cause := tmp.Error()
+ if stringsutil.ContainsAll(cause, "ReadStatusLine:", "read: connection reset by peer") {
+ // this is a FP and should not be counted as a host error
+ // because server closes connection when it reads corrupted bytes which we send via rawhttp
+ return false
+ }
+ if strings.HasPrefix(cause, "ReadStatusLine:") {
+ // error is present in last part when using rawhttp
+ // this will be fixed once errkit is used everywhere
+ lastIndex := strings.LastIndex(cause, ":")
+ if lastIndex == -1 {
+ lastIndex = 0
+ }
+ if lastIndex >= len(cause)-1 {
+ lastIndex = 0
+ }
+ cause = cause[lastIndex+1:]
+ }
+ for _, msg := range c.TrackError {
+ if strings.Contains(cause, msg) {
+ return true
+ }
}
+ return reCheckError.MatchString(cause)
}
- return reCheckError.MatchString(errString)
}
diff --git a/pkg/protocols/common/hosterrorscache/hosterrorscache_test.go b/pkg/protocols/common/hosterrorscache/hosterrorscache_test.go
index a4730366e0..9977b968d9 100644
--- a/pkg/protocols/common/hosterrorscache/hosterrorscache_test.go
+++ b/pkg/protocols/common/hosterrorscache/hosterrorscache_test.go
@@ -1,20 +1,26 @@
package hosterrorscache
import (
+ "context"
"fmt"
"sync"
"sync/atomic"
"testing"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/stretchr/testify/require"
)
+const (
+ protoType = "http"
+)
+
func TestCacheCheck(t *testing.T) {
cache := New(3, DefaultMaxHostsCount, nil)
for i := 0; i < 100; i++ {
- cache.MarkFailed("test", fmt.Errorf("could not resolve host"))
- got := cache.Check("test")
+ cache.MarkFailed(protoType, newCtxArgs("test"), fmt.Errorf("could not resolve host"))
+ got := cache.Check(protoType, newCtxArgs("test"))
if i < 2 {
// till 3 the host is not flagged to skip
require.False(t, got)
@@ -24,7 +30,7 @@ func TestCacheCheck(t *testing.T) {
}
}
- value := cache.Check("test")
+ value := cache.Check(protoType, newCtxArgs("test"))
require.Equal(t, true, value, "could not get checked value")
}
@@ -32,8 +38,8 @@ func TestTrackErrors(t *testing.T) {
cache := New(3, DefaultMaxHostsCount, []string{"custom error"})
for i := 0; i < 100; i++ {
- cache.MarkFailed("custom", fmt.Errorf("got: nested: custom error"))
- got := cache.Check("custom")
+ cache.MarkFailed(protoType, newCtxArgs("custom"), fmt.Errorf("got: nested: custom error"))
+ got := cache.Check(protoType, newCtxArgs("custom"))
if i < 2 {
// till 3 the host is not flagged to skip
require.False(t, got)
@@ -42,7 +48,7 @@ func TestTrackErrors(t *testing.T) {
require.True(t, got)
}
}
- value := cache.Check("custom")
+ value := cache.Check(protoType, newCtxArgs("custom"))
require.Equal(t, true, value, "could not get checked value")
}
@@ -73,23 +79,23 @@ func TestCacheMarkFailed(t *testing.T) {
tests := []struct {
host string
- expected int
+ expected int32
}{
{"http://example.com:80", 1},
{"example.com:80", 2},
- {"example.com", 1},
+ // earlier if port is not provided then port was omitted
+ // but from now it will default to appropriate http scheme based port with 80 as default
+ {"example.com:443", 1},
}
for _, test := range tests {
- normalizedCacheValue := cache.normalizeCacheValue(test.host)
- cache.MarkFailed(test.host, fmt.Errorf("no address found for host"))
+ normalizedCacheValue := cache.GetKeyFromContext(newCtxArgs(test.host), nil)
+ cache.MarkFailed(protoType, newCtxArgs(test.host), fmt.Errorf("no address found for host"))
failedTarget, err := cache.failedTargets.Get(normalizedCacheValue)
require.Nil(t, err)
require.NotNil(t, failedTarget)
- value, ok := failedTarget.(*cacheItem)
- require.True(t, ok)
- require.EqualValues(t, test.expected, value.errors.Load())
+ require.EqualValues(t, test.expected, failedTarget.errors.Load())
}
}
@@ -102,12 +108,12 @@ func TestCacheMarkFailedConcurrent(t *testing.T) {
}{
{"http://example.com:80", 200},
{"example.com:80", 200},
- {"example.com", 100},
+ {"example.com:443", 100},
}
// the cache is not atomic during items creation, so we pre-create them with counter to zero
for _, test := range tests {
- normalizedValue := cache.normalizeCacheValue(test.host)
+ normalizedValue := cache.NormalizeCacheValue(test.host)
newItem := &cacheItem{errors: atomic.Int32{}}
newItem.errors.Store(0)
_ = cache.failedTargets.Set(normalizedValue, newItem)
@@ -120,22 +126,25 @@ func TestCacheMarkFailedConcurrent(t *testing.T) {
wg.Add(1)
go func() {
defer wg.Done()
- cache.MarkFailed(currentTest.host, fmt.Errorf("could not resolve host"))
+ cache.MarkFailed(protoType, newCtxArgs(currentTest.host), fmt.Errorf("could not resolve host"))
}()
}
}
wg.Wait()
for _, test := range tests {
- require.True(t, cache.Check(test.host))
+ require.True(t, cache.Check(protoType, newCtxArgs(test.host)))
- normalizedCacheValue := cache.normalizeCacheValue(test.host)
+ normalizedCacheValue := cache.NormalizeCacheValue(test.host)
failedTarget, err := cache.failedTargets.Get(normalizedCacheValue)
require.Nil(t, err)
require.NotNil(t, failedTarget)
- value, ok := failedTarget.(*cacheItem)
- require.True(t, ok)
- require.EqualValues(t, test.expected, value.errors.Load())
+ require.EqualValues(t, test.expected, failedTarget.errors.Load())
}
}
+
+func newCtxArgs(value string) *contextargs.Context {
+ ctx := contextargs.NewWithInput(context.TODO(), value)
+ return ctx
+}
diff --git a/pkg/protocols/common/interactsh/const.go b/pkg/protocols/common/interactsh/const.go
index 079940240f..aad130d468 100644
--- a/pkg/protocols/common/interactsh/const.go
+++ b/pkg/protocols/common/interactsh/const.go
@@ -8,7 +8,7 @@ import (
var (
defaultInteractionDuration = 60 * time.Second
- interactshURLMarkerRegex = regexp.MustCompile(`{{interactsh-url(?:_[0-9]+){0,3}}}`)
+ interactshURLMarkerRegex = regexp.MustCompile(`(%7[B|b]|\{){2}(interactsh-url(?:_[0-9]+){0,3})(%7[D|d]|\}){2}`)
ErrInteractshClientNotInitialized = errors.New("interactsh client not initialized")
)
diff --git a/pkg/protocols/common/interactsh/interactsh.go b/pkg/protocols/common/interactsh/interactsh.go
index 0b38af041c..f538f58016 100644
--- a/pkg/protocols/common/interactsh/interactsh.go
+++ b/pkg/protocols/common/interactsh/interactsh.go
@@ -21,6 +21,21 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/interactsh/pkg/client"
"github.com/projectdiscovery/interactsh/pkg/server"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/interactsh/interactsh.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/writer"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/writer"
+ "github.com/projectdiscovery/retryablehttp-go"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/interactsh/interactsh.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -180,6 +195,14 @@ func (c *Client) processInteractionForRequest(interaction *server.Interaction, d
gologger.DefaultLogger.Print().Msgf("[Interactsh]: got result %v and status %v after processing interaction", result, matched)
}
+ if c.options.FuzzParamsFrequency != nil {
+ if !matched {
+ c.options.FuzzParamsFrequency.MarkParameter(data.Parameter, data.Request.URL.String(), data.Operators.TemplateID)
+ } else {
+ c.options.FuzzParamsFrequency.UnmarkParameter(data.Parameter, data.Request.URL.String(), data.Operators.TemplateID)
+ }
+ }
+
// if we don't match, return
if !matched || result == nil {
return false
@@ -320,6 +343,9 @@ type RequestData struct {
Operators *operators.Operators
MatchFunc operators.MatchFunc
ExtractFunc operators.ExtractFunc
+
+ Parameter string
+ Request *retryablehttp.Request
}
// RequestEvent is the event for a network request sent by nuclei.
diff --git a/pkg/protocols/common/interactsh/options.go b/pkg/protocols/common/interactsh/options.go
index a42b0dccb9..0646321f7f 100644
--- a/pkg/protocols/common/interactsh/options.go
+++ b/pkg/protocols/common/interactsh/options.go
@@ -7,6 +7,19 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/progress"
"github.com/Explorer1092/nuclei/v3/pkg/reporting"
"github.com/projectdiscovery/interactsh/pkg/client"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/interactsh/options.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/progress"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz/frequency"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/progress"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/interactsh/options.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
)
@@ -46,8 +59,9 @@ type Options struct {
// NoColor disables printing colors for matches
NoColor bool
- StopAtFirstMatch bool
- HTTPClient *retryablehttp.Client
+ FuzzParamsFrequency *frequency.Tracker
+ StopAtFirstMatch bool
+ HTTPClient *retryablehttp.Client
}
// DefaultOptions returns the default options for interactsh client
diff --git a/pkg/protocols/common/protocolinit/init.go b/pkg/protocols/common/protocolinit/init.go
index c7dc8630a0..1bdde09f91 100644
--- a/pkg/protocols/common/protocolinit/init.go
+++ b/pkg/protocols/common/protocolinit/init.go
@@ -13,7 +13,6 @@ import (
// Init initializes the client pools for the protocols
func Init(options *types.Options) error {
-
if err := protocolstate.Init(options); err != nil {
return err
}
@@ -39,5 +38,5 @@ func Init(options *types.Options) error {
}
func Close() {
- protocolstate.Dialer.Close()
+ protocolstate.Close()
}
diff --git a/pkg/protocols/common/protocolstate/memguardian.go b/pkg/protocols/common/protocolstate/memguardian.go
index 1db1e0a775..dac57cb7ed 100644
--- a/pkg/protocols/common/protocolstate/memguardian.go
+++ b/pkg/protocols/common/protocolstate/memguardian.go
@@ -19,7 +19,7 @@ var (
)
func StartActiveMemGuardian(ctx context.Context) {
- if memguardian.DefaultMemGuardian == nil {
+ if memguardian.DefaultMemGuardian == nil || memTimer != nil {
return
}
@@ -77,9 +77,8 @@ var muGlobalChange sync.Mutex
// Global setting
func GlobalGuardBytesBufferAlloc() error {
- if muGlobalChange.TryLock() {
+ if !muGlobalChange.TryLock() {
return nil
-
}
defer muGlobalChange.Unlock()
@@ -95,9 +94,8 @@ func GlobalGuardBytesBufferAlloc() error {
// Global setting
func GlobalRestoreBytesBufferAlloc() {
- if muGlobalChange.TryLock() {
+ if !muGlobalChange.TryLock() {
return
-
}
defer muGlobalChange.Unlock()
diff --git a/pkg/protocols/common/protocolstate/state.go b/pkg/protocols/common/protocolstate/state.go
index 167ababda0..a4c6e31392 100644
--- a/pkg/protocols/common/protocolstate/state.go
+++ b/pkg/protocols/common/protocolstate/state.go
@@ -5,6 +5,7 @@ import (
"fmt"
"net"
"net/url"
+ "sync"
"github.com/go-sql-driver/mysql"
"github.com/pkg/errors"
@@ -15,13 +16,34 @@ import (
"github.com/projectdiscovery/fastdialer/fastdialer"
"github.com/projectdiscovery/mapcidr/asn"
"github.com/projectdiscovery/networkpolicy"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/protocolstate/state.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/expand"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/protocolstate/state.go
+>>>>>>> projectdiscovery-main
)
// Dialer is a shared fastdialer instance for host DNS resolution
var (
- Dialer *fastdialer.Dialer
+ muDialer sync.RWMutex
+ Dialer *fastdialer.Dialer
)
+func GetDialer() *fastdialer.Dialer {
+ muDialer.RLock()
+ defer muDialer.RUnlock()
+
+ return Dialer
+}
+
+func ShouldInit() bool {
+ return Dialer == nil
+}
+
// Init creates the Dialer instance based on user configuration
func Init(options *types.Options) error {
if Dialer != nil {
@@ -30,9 +52,7 @@ func Init(options *types.Options) error {
lfaAllowed = options.AllowLocalFileAccess
opts := fastdialer.DefaultOptions
- if options.DialerTimeout > 0 {
- opts.DialerTimeout = options.DialerTimeout
- }
+ opts.DialerTimeout = options.GetTimeouts().DialTimeout
if options.DialerKeepAlive > 0 {
opts.DialerKeepAlive = options.DialerKeepAlive
}
@@ -180,6 +200,7 @@ func interfaceAddress(interfaceName string) (net.IP, error) {
if ipnet, ok := addr.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {
if ipnet.IP.To4() != nil {
address = ipnet.IP
+ break
}
}
}
@@ -207,8 +228,12 @@ func interfaceAddresses(interfaceName string) ([]net.Addr, error) {
// Close closes the global shared fastdialer
func Close() {
+ muDialer.Lock()
+ defer muDialer.Unlock()
+
if Dialer != nil {
Dialer.Close()
+ Dialer = nil
}
StopActiveMemGuardian()
}
diff --git a/pkg/protocols/common/replacer/replacer.go b/pkg/protocols/common/replacer/replacer.go
index bb28d01977..fe96f9f9a3 100644
--- a/pkg/protocols/common/replacer/replacer.go
+++ b/pkg/protocols/common/replacer/replacer.go
@@ -5,8 +5,18 @@ import (
"github.com/projectdiscovery/fasttemplate"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/marker"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/replacer/replacer.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/marker"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/marker"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/replacer/replacer.go
+>>>>>>> projectdiscovery-main
)
// Replace replaces placeholders in template with values on the fly.
diff --git a/pkg/protocols/common/uncover/uncover.go b/pkg/protocols/common/uncover/uncover.go
index 8276364e2c..d9f971fc79 100644
--- a/pkg/protocols/common/uncover/uncover.go
+++ b/pkg/protocols/common/uncover/uncover.go
@@ -8,6 +8,29 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/templates"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/uncover/uncover.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/templates"
+ "github.com/projectdiscovery/ratelimit"
+ ucRunner "github.com/projectdiscovery/uncover/runner"
+ "github.com/projectdiscovery/uncover/uncover"
+ "github.com/projectdiscovery/uncover/uncover/agent/censys"
+ "github.com/projectdiscovery/uncover/uncover/agent/criminalip"
+ "github.com/projectdiscovery/uncover/uncover/agent/fofa"
+ "github.com/projectdiscovery/uncover/uncover/agent/hunter"
+ "github.com/projectdiscovery/uncover/uncover/agent/netlas"
+ "github.com/projectdiscovery/uncover/uncover/agent/quake"
+ "github.com/projectdiscovery/uncover/uncover/agent/shodan"
+ "github.com/projectdiscovery/uncover/uncover/agent/shodanidb"
+ "github.com/projectdiscovery/uncover/uncover/agent/zoomeye"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/templates"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/uncover/uncover.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/uncover"
"github.com/projectdiscovery/uncover/sources"
mapsutil "github.com/projectdiscovery/utils/maps"
diff --git a/pkg/protocols/common/utils/vardump/dump.go b/pkg/protocols/common/utils/vardump/dump.go
index f36bc1d061..eb6b5b2c09 100644
--- a/pkg/protocols/common/utils/vardump/dump.go
+++ b/pkg/protocols/common/utils/vardump/dump.go
@@ -1,53 +1,72 @@
package vardump
import (
- "strconv"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
mapsutil "github.com/projectdiscovery/utils/maps"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ mapsutil "github.com/projectdiscovery/utils/maps"
+ "github.com/yassinebenaid/godump"
+>>>>>>> projectdiscovery-main
)
-// EnableVarDump enables var dump for debugging optionally
-var EnableVarDump bool
+// variables is a map of variables
+type variables = map[string]any
+
+// DumpVariables dumps the variables in a pretty format
+func DumpVariables(data variables) string {
+ if !EnableVarDump {
+ return ""
+ }
-// DumpVariables writes the truncated dump of variables to a string
-// in a formatted key-value manner.
-//
-// The values are truncated to return 50 characters from start and end.
-func DumpVariables(data map[string]interface{}) string {
- var counter int
+ d := godump.Dumper{
+ Indentation: " ",
+ HidePrivateFields: false,
+ ShowPrimitiveNamedTypes: true,
+ }
- buffer := &strings.Builder{}
- buffer.Grow(len(data) * 78) // grow buffer to an approximate size
+ d.Theme = godump.Theme{
+ String: godump.RGB{R: 138, G: 201, B: 38},
+ Quotes: godump.RGB{R: 112, G: 214, B: 255},
+ Bool: godump.RGB{R: 249, G: 87, B: 56},
+ Number: godump.RGB{R: 10, G: 178, B: 242},
+ Types: godump.RGB{R: 0, G: 150, B: 199},
+ Address: godump.RGB{R: 205, G: 93, B: 0},
+ PointerTag: godump.RGB{R: 110, G: 110, B: 110},
+ Nil: godump.RGB{R: 219, G: 57, B: 26},
+ Func: godump.RGB{R: 160, G: 90, B: 220},
+ Fields: godump.RGB{R: 189, G: 176, B: 194},
+ Chan: godump.RGB{R: 195, G: 154, B: 76},
+ UnsafePointer: godump.RGB{R: 89, G: 193, B: 180},
+ Braces: godump.RGB{R: 185, G: 86, B: 86},
+ }
- builder := &strings.Builder{}
- // sort keys for deterministic output
+ return d.Sprint(process(data, Limit))
+}
+
+// process is a helper function that processes the variables
+// and returns a new map of variables
+func process(data variables, limit int) variables {
keys := mapsutil.GetSortedKeys(data)
+ vars := make(variables)
+
+ if limit == 0 {
+ limit = 255
+ }
for _, k := range keys {
- v := data[k]
- valueString := types.ToString(v)
-
- counter++
- if len(valueString) > 50 {
- builder.Grow(56)
- builder.WriteString(valueString[0:25])
- builder.WriteString(" .... ")
- builder.WriteString(valueString[len(valueString)-25:])
- valueString = builder.String()
- builder.Reset()
+ v := types.ToString(data[k])
+ v = strings.ReplaceAll(strings.ReplaceAll(v, "\r", " "), "\n", " ")
+ if len(v) > limit {
+ v = v[:limit]
+ v += " [...]"
}
- valueString = strings.ReplaceAll(strings.ReplaceAll(valueString, "\r", " "), "\n", " ")
-
- buffer.WriteString("\t")
- buffer.WriteString(strconv.Itoa(counter))
- buffer.WriteString(". ")
- buffer.WriteString(k)
- buffer.WriteString(" => ")
- buffer.WriteString(valueString)
- buffer.WriteString("\n")
+
+ vars[k] = v
}
- final := buffer.String()
- return final
+
+ return vars
}
diff --git a/pkg/protocols/common/utils/vardump/dump_test.go b/pkg/protocols/common/utils/vardump/dump_test.go
new file mode 100644
index 0000000000..9929fa5318
--- /dev/null
+++ b/pkg/protocols/common/utils/vardump/dump_test.go
@@ -0,0 +1,55 @@
+package vardump
+
+import (
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+)
+
+func TestDumpVariables(t *testing.T) {
+ // Enable var dump for testing
+ EnableVarDump = true
+
+ // Test case
+ testVars := variables{
+ "string": "test",
+ "int": 42,
+ "bool": true,
+ "slice": []string{"a", "b", "c"},
+ }
+
+ result := DumpVariables(testVars)
+
+ // Assertions
+ assert.NotEmpty(t, result)
+ assert.Contains(t, result, "string")
+ assert.Contains(t, result, "test")
+ assert.Contains(t, result, "int")
+ assert.Contains(t, result, "42")
+ assert.Contains(t, result, "bool")
+ assert.Contains(t, result, "true")
+ assert.Contains(t, result, "slice")
+ assert.Contains(t, result, "a")
+ assert.Contains(t, result, "b")
+ assert.Contains(t, result, "c")
+
+ // Test with EnableVarDump set to false
+ EnableVarDump = false
+ result = DumpVariables(testVars)
+ assert.Empty(t, result)
+}
+
+func TestProcess(t *testing.T) {
+ testVars := variables{
+ "short": "short string",
+ "long": strings.Repeat("a", 300),
+ "number": 42,
+ }
+
+ processed := process(testVars, 255)
+
+ assert.Equal(t, "short string", processed["short"])
+ assert.Equal(t, strings.Repeat("a", 255)+" [...]", processed["long"])
+ assert.Equal(t, "42", processed["number"])
+}
diff --git a/pkg/protocols/common/utils/vardump/vars.go b/pkg/protocols/common/utils/vardump/vars.go
new file mode 100644
index 0000000000..f5e18bce69
--- /dev/null
+++ b/pkg/protocols/common/utils/vardump/vars.go
@@ -0,0 +1,8 @@
+package vardump
+
+var (
+ // EnableVarDump enables var dump for debugging optionally
+ EnableVarDump bool
+ // Limit is the maximum characters to be dumped
+ Limit int = 255
+)
diff --git a/pkg/protocols/common/variables/variables.go b/pkg/protocols/common/variables/variables.go
index 31aec1368c..71d0765679 100644
--- a/pkg/protocols/common/variables/variables.go
+++ b/pkg/protocols/common/variables/variables.go
@@ -4,6 +4,7 @@ import (
"encoding/json"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/expressions"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/interactsh"
@@ -11,6 +12,33 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/Explorer1092/nuclei/v3/pkg/utils"
"github.com/invopop/jsonschema"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/common/variables/variables.go
+ "github.com/alecthomas/jsonschema"
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/interactsh"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/marker"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+ "github.com/Explorer1092/nuclei/v2/pkg/utils"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/interactsh"
+ protocolutils "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+ "github.com/projectdiscovery/nuclei/v2/pkg/utils"
+=======
+ "github.com/invopop/jsonschema"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/common/variables/variables.go
+>>>>>>> projectdiscovery-main
stringsutil "github.com/projectdiscovery/utils/strings"
)
diff --git a/pkg/protocols/dns/cluster.go b/pkg/protocols/dns/cluster.go
index f2f5734785..86852b4d78 100644
--- a/pkg/protocols/dns/cluster.go
+++ b/pkg/protocols/dns/cluster.go
@@ -1,27 +1,24 @@
package dns
-// CanCluster returns true if the request can be clustered.
-//
-// This used by the clustering engine to decide whether two requests
-// are similar enough to be considered one and can be checked by
-// just adding the matcher/extractors for the request and the correct IDs.
-func (request *Request) CanCluster(other *Request) bool {
- if len(request.Resolvers) > 0 || request.Trace || request.ID != "" {
- return false
- }
- if request.Name != other.Name ||
- request.class != other.class ||
- request.Retries != other.Retries ||
- request.question != other.question {
- return false
- }
+import (
+ "fmt"
+
+ "github.com/cespare/xxhash"
+)
+
+
+// TmplClusterKey generates a unique key for the request
+// to be used in the clustering process.
+func (request *Request) TmplClusterKey() uint64 {
+ recursion := ""
if request.Recursion != nil {
- if other.Recursion == nil {
- return false
- }
- if *request.Recursion != *other.Recursion {
- return false
- }
+ recursion = fmt.Sprintf("%t", *request.Recursion)
}
- return true
+ inp := fmt.Sprintf("%s-%d-%d-%d-%s", request.Name, request.class, request.Retries, request.question, recursion)
+ return xxhash.Sum64String(inp)
+}
+
+// IsClusterable returns true if the request is eligible to be clustered.
+func (request *Request) IsClusterable() bool {
+ return !(len(request.Resolvers) > 0 || request.Trace || request.ID != "")
}
diff --git a/pkg/protocols/dns/dns.go b/pkg/protocols/dns/dns.go
index 2ff933cbc9..e4daf43959 100644
--- a/pkg/protocols/dns/dns.go
+++ b/pkg/protocols/dns/dns.go
@@ -6,12 +6,40 @@ import (
"github.com/miekg/dns"
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/expressions"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/replacer"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/dns/dnsclientpool"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/dns/dns.go
+<<<<<<< HEAD
+ "github.com/weppos/publicsuffix-go/publicsuffix"
+
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/replacer"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/dns/dnsclientpool"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/replacer"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/dns/dnsclientpool"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/replacer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/dns/dnsclientpool"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/dns/dns.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryabledns"
fileutil "github.com/projectdiscovery/utils/file"
)
@@ -141,12 +169,6 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
recursion := true
request.Recursion = &recursion
}
- dnsClientOptions := &dnsclientpool.Configuration{
- Retries: request.Retries,
- }
- if len(request.Resolvers) > 0 {
- dnsClientOptions.Resolvers = request.Resolvers
- }
// Create a dns client for the class
client, err := request.getDnsClient(options, nil)
if err != nil {
diff --git a/pkg/protocols/dns/dns_test.go b/pkg/protocols/dns/dns_test.go
index dcd0b7c753..a4c905d249 100644
--- a/pkg/protocols/dns/dns_test.go
+++ b/pkg/protocols/dns/dns_test.go
@@ -5,9 +5,21 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/dns/dns_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/dns/dns_test.go
+>>>>>>> projectdiscovery-main
)
func TestDNSCompileMake(t *testing.T) {
diff --git a/pkg/protocols/dns/dnsclientpool/clientpool.go b/pkg/protocols/dns/dnsclientpool/clientpool.go
index 48cc4da461..1172347850 100644
--- a/pkg/protocols/dns/dnsclientpool/clientpool.go
+++ b/pkg/protocols/dns/dnsclientpool/clientpool.go
@@ -7,6 +7,14 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/pkg/errors"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/dns/dnsclientpool/clientpool.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/dns/dnsclientpool/clientpool.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryabledns"
)
diff --git a/pkg/protocols/dns/operators.go b/pkg/protocols/dns/operators.go
index 178f1110c4..189ed195ac 100644
--- a/pkg/protocols/dns/operators.go
+++ b/pkg/protocols/dns/operators.go
@@ -8,6 +8,7 @@ import (
"github.com/miekg/dns"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/operators/extractors"
"github.com/Explorer1092/nuclei/v3/pkg/operators/matchers"
@@ -15,6 +16,34 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/dns/operators.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/model"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v2/pkg/output"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/dns/operators.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryabledns"
)
@@ -114,6 +143,7 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(wrapped.InternalEvent["template-id"]),
TemplatePath: types.ToString(wrapped.InternalEvent["template-path"]),
Info: wrapped.InternalEvent["template-info"].(model.Info),
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Host: types.ToString(wrapped.InternalEvent["host"]),
Matched: types.ToString(wrapped.InternalEvent["matched"]),
diff --git a/pkg/protocols/dns/operators_test.go b/pkg/protocols/dns/operators_test.go
index 0da1485891..103cc5815d 100644
--- a/pkg/protocols/dns/operators_test.go
+++ b/pkg/protocols/dns/operators_test.go
@@ -8,6 +8,7 @@ import (
"github.com/miekg/dns"
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
@@ -15,6 +16,25 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/operators/matchers"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/dns/operators_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/dns/operators_test.go
+>>>>>>> projectdiscovery-main
)
func TestResponseToDSLMap(t *testing.T) {
diff --git a/pkg/protocols/dns/request.go b/pkg/protocols/dns/request.go
index 4c551fe419..8c90a984a3 100644
--- a/pkg/protocols/dns/request.go
+++ b/pkg/protocols/dns/request.go
@@ -3,7 +3,6 @@ package dns
import (
"encoding/hex"
"fmt"
- "net/url"
"strings"
"sync"
@@ -24,6 +23,46 @@ import (
templateTypes "github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/Explorer1092/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/dns/request.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/utils/vardump"
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+ "github.com/Explorer1092/nuclei/v2/pkg/utils"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/output"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/utils/vardump"
+ protocolutils "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v2/pkg/utils"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/dns/request.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryabledns"
iputil "github.com/projectdiscovery/utils/ip"
syncutil "github.com/projectdiscovery/utils/sync"
@@ -38,16 +77,8 @@ func (request *Request) Type() templateTypes.ProtocolType {
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (request *Request) ExecuteWithResults(input *contextargs.Context, metadata, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
- // Parse the URL and return domain if URL.
- var domain string
- if utils.IsURL(input.MetaInput.Input) {
- domain = extractDomain(input.MetaInput.Input)
- } else {
- domain = input.MetaInput.Input
- }
-
var err error
- domain, err = request.parseDNSInput(domain)
+ domain, err := request.parseDNSInput(input.MetaInput.Input)
if err != nil {
return errors.Wrap(err, "could not build request")
}
@@ -116,9 +147,9 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, metadata,
}
func (request *Request) execute(input *contextargs.Context, domain string, metadata, previous output.InternalEvent, vars map[string]interface{}, callback protocols.OutputEventCallback) error {
-
+ var err error
if vardump.EnableVarDump {
- gologger.Debug().Msgf("DNS Protocol request variables: \n%s\n", vardump.DumpVariables(vars))
+ gologger.Debug().Msgf("DNS Protocol request variables: %s\n", vardump.DumpVariables(vars))
}
// Compile each request for the template based on the URL
@@ -152,7 +183,7 @@ func (request *Request) execute(input *contextargs.Context, domain string, metad
if request.options.Options.Debug || request.options.Options.DebugRequests || request.options.Options.StoreResponse {
msg := fmt.Sprintf("[%s] Dumped DNS request for %s", request.options.TemplateID, question)
if request.options.Options.Debug || request.options.Options.DebugRequests {
- gologger.Info().Str("domain", domain).Msgf(msg)
+ gologger.Info().Str("domain", domain).Msg(msg)
gologger.Print().Msgf("%s", requestString)
}
if request.options.Options.StoreResponse {
@@ -209,7 +240,7 @@ func (request *Request) execute(input *contextargs.Context, domain string, metad
}
callback(event)
- return nil
+ return err
}
func (request *Request) parseDNSInput(host string) (string, error) {
@@ -230,7 +261,7 @@ func (request *Request) parseDNSInput(host string) (string, error) {
return host, nil
}
-func dumpResponse(event *output.InternalWrappedEvent, request *Request, requestOptions *protocols.ExecutorOptions, response, domain string) {
+func dumpResponse(event *output.InternalWrappedEvent, request *Request, _ *protocols.ExecutorOptions, response, domain string) {
cliOptions := request.options.Options
if cliOptions.Debug || cliOptions.DebugResponse || cliOptions.StoreResponse {
hexDump := false
@@ -261,12 +292,3 @@ func dumpTraceData(event *output.InternalWrappedEvent, requestOptions *protocols
gologger.Debug().Msgf("[%s] Dumped DNS Trace data for %s\n\n%s", requestOptions.TemplateID, domain, highlightedResponse)
}
}
-
-// extractDomain extracts the domain name of a URL
-func extractDomain(theURL string) string {
- u, err := url.Parse(theURL)
- if err != nil {
- return ""
- }
- return u.Hostname()
-}
diff --git a/pkg/protocols/dns/request_test.go b/pkg/protocols/dns/request_test.go
index 9454e7651f..a84f813d80 100644
--- a/pkg/protocols/dns/request_test.go
+++ b/pkg/protocols/dns/request_test.go
@@ -6,6 +6,7 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
@@ -14,6 +15,27 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/dns/request_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/dns/request_test.go
+>>>>>>> projectdiscovery-main
)
func TestDNSExecuteWithResults(t *testing.T) {
@@ -67,19 +89,5 @@ func TestDNSExecuteWithResults(t *testing.T) {
require.Equal(t, 1, len(finalEvent.Results[0].ExtractedResults), "could not get correct number of extracted results")
require.Equal(t, "93.184.215.14", finalEvent.Results[0].ExtractedResults[0], "could not get correct extracted results")
finalEvent = nil
-
- t.Run("url-to-domain", func(t *testing.T) {
- metadata := make(output.InternalEvent)
- previous := make(output.InternalEvent)
- err := request.ExecuteWithResults(contextargs.NewWithInput(context.Background(), "https://example.com"), metadata, previous, func(event *output.InternalWrappedEvent) {
- finalEvent = event
- })
- require.Nil(t, err, "could not execute dns request")
- })
- require.NotNil(t, finalEvent, "could not get event output from request")
- require.Equal(t, 1, len(finalEvent.Results), "could not get correct number of results")
- require.Equal(t, "test", finalEvent.Results[0].MatcherName, "could not get correct matcher name of results")
- require.Equal(t, 1, len(finalEvent.Results[0].ExtractedResults), "could not get correct number of extracted results")
- require.Equal(t, "93.184.215.14", finalEvent.Results[0].ExtractedResults[0], "could not get correct extracted results")
- finalEvent = nil
+ // Note: changing url to domain is responsible at tmplexec package and is implemented there
}
diff --git a/pkg/protocols/file/file.go b/pkg/protocols/file/file.go
index 4a3dbd8303..283a43ce80 100644
--- a/pkg/protocols/file/file.go
+++ b/pkg/protocols/file/file.go
@@ -8,8 +8,18 @@ import (
"github.com/h2non/filetype"
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/file/file.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/file/file.go
+>>>>>>> projectdiscovery-main
)
var (
diff --git a/pkg/protocols/file/find_test.go b/pkg/protocols/file/find_test.go
index 1061cf8070..cd1fefc1a5 100644
--- a/pkg/protocols/file/find_test.go
+++ b/pkg/protocols/file/find_test.go
@@ -7,10 +7,23 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
permissionutil "github.com/projectdiscovery/utils/permission"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/file/find_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+ permissionutil "github.com/projectdiscovery/utils/permission"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/file/find_test.go
+>>>>>>> projectdiscovery-main
)
func TestFindInputPaths(t *testing.T) {
diff --git a/pkg/protocols/file/operators.go b/pkg/protocols/file/operators.go
index bd4a5a4aea..6cc266f4c0 100644
--- a/pkg/protocols/file/operators.go
+++ b/pkg/protocols/file/operators.go
@@ -3,6 +3,7 @@ package file
import (
"time"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/operators/extractors"
@@ -10,6 +11,25 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/file/operators.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/file/operators.go
+>>>>>>> projectdiscovery-main
)
// Match matches a generic data response again a given matcher
@@ -104,6 +124,7 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(wrapped.InternalEvent["template-id"]),
TemplatePath: types.ToString(wrapped.InternalEvent["template-path"]),
Info: wrapped.InternalEvent["template-info"].(model.Info),
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Path: types.ToString(wrapped.InternalEvent["path"]),
Matched: types.ToString(wrapped.InternalEvent["matched"]),
diff --git a/pkg/protocols/file/operators_test.go b/pkg/protocols/file/operators_test.go
index 556c1cf8c4..0465cee46f 100644
--- a/pkg/protocols/file/operators_test.go
+++ b/pkg/protocols/file/operators_test.go
@@ -5,6 +5,7 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
@@ -12,6 +13,25 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/operators/matchers"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/file/operators_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/file/operators_test.go
+>>>>>>> projectdiscovery-main
)
func newMockOperator() operators.Operators {
diff --git a/pkg/protocols/file/request.go b/pkg/protocols/file/request.go
index 8c6604bdb3..9cdb59403f 100644
--- a/pkg/protocols/file/request.go
+++ b/pkg/protocols/file/request.go
@@ -22,6 +22,29 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
templateTypes "github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/file/request.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/file/request.go
+>>>>>>> projectdiscovery-main
sliceutil "github.com/projectdiscovery/utils/slice"
syncutil "github.com/projectdiscovery/utils/sync"
)
@@ -51,6 +74,9 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, metadata,
if err != nil {
return err
}
+ if input.MetaInput.Input == "" {
+ return errors.New("input cannot be empty file or folder expected")
+ }
err = request.getInputPaths(input.MetaInput.Input, func(filePath string) {
wg.Add()
func(filePath string) {
@@ -250,6 +276,8 @@ func (request *Request) findMatchesWithReader(reader io.Reader, input *contextar
for k, v := range previous {
dslMap[k] = v
}
+ // add vars to template context
+ request.options.AddTemplateVars(input.MetaInput, request.Type(), request.ID, dslMap)
// add template context variables to DSL map
if request.options.HasTemplateCtx(input.MetaInput) {
dslMap = generators.MergeMaps(dslMap, request.options.GetTemplateCtx(input.MetaInput).GetAll())
@@ -323,7 +351,6 @@ func (request *Request) buildEvent(input, filePath string, fileMatches []FileMat
exprLines[fileMatch.Expr] = append(exprLines[fileMatch.Expr], fileMatch.Line)
exprBytes[fileMatch.Expr] = append(exprBytes[fileMatch.Expr], fileMatch.ByteIndex)
}
-
event := eventcreator.CreateEventWithOperatorResults(request, internalEvent, operatorResult)
// Annotate with line numbers if asked by the user
if request.options.Options.ShowMatchLine {
diff --git a/pkg/protocols/file/request_test.go b/pkg/protocols/file/request_test.go
index be86900e4d..15e01441e5 100644
--- a/pkg/protocols/file/request_test.go
+++ b/pkg/protocols/file/request_test.go
@@ -8,6 +8,7 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
@@ -17,6 +18,28 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
permissionutil "github.com/projectdiscovery/utils/permission"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/file/request_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+ permissionutil "github.com/projectdiscovery/utils/permission"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/file/request_test.go
+>>>>>>> projectdiscovery-main
)
func TestFileExecuteWithResults(t *testing.T) {
diff --git a/pkg/protocols/headless/engine/action_types.go b/pkg/protocols/headless/engine/action_types.go
index c9cd39ad2a..d66fe91130 100644
--- a/pkg/protocols/headless/engine/action_types.go
+++ b/pkg/protocols/headless/engine/action_types.go
@@ -6,11 +6,15 @@ import (
"strings"
"github.com/invopop/jsonschema"
+ mapsutil "github.com/projectdiscovery/utils/maps"
)
// ActionType defines the action type for a browser action
type ActionType int8
+// ActionData stores the action output data
+type ActionData = mapsutil.Map[string, any]
+
// Types to be executed by the user.
// name:ActionType
const (
@@ -41,9 +45,24 @@ const (
// ActionFilesInput performs an action on a file input.
// name:files
ActionFilesInput
- // ActionWaitLoad waits for the page to stop loading.
+ // ActionWaitDOM waits for the HTML document has been completely loaded & parsed.
+ // name:waitdom
+ ActionWaitDOM
+ // ActionWaitFCP waits for the first piece of content (text, image, etc.) is painted on the screen.
+ // name:waitfcp
+ ActionWaitFCP
+ // ActionWaitFMP waits for page has rendered enough meaningful content to be useful to the user.
+ // name:waitfmp
+ ActionWaitFMP
+ // ActionWaitIdle waits for the network is completely idle (no ongoing network requests).
+ // name:waitidle
+ ActionWaitIdle
+ // ActionWaitLoad waits for the page and all its resources (like stylesheets and images) have finished loading.
// name:waitload
ActionWaitLoad
+ // ActionWaitStable waits until the page is stable.
+ // name:waitstable
+ ActionWaitStable
// ActionGetResource performs a get resource action on an element
// name:getresource
ActionGetResource
@@ -68,6 +87,9 @@ const (
// ActionWaitEvent waits for a specific event.
// name:waitevent
ActionWaitEvent
+ // ActionWaitDialog waits for JavaScript dialog (alert, confirm, prompt, or onbeforeunload).
+ // name:dialog
+ ActionWaitDialog
// ActionKeyboard performs a keyboard action event on a page.
// name:keyboard
ActionKeyboard
@@ -95,7 +117,12 @@ var ActionStringToAction = map[string]ActionType{
"time": ActionTimeInput,
"select": ActionSelectInput,
"files": ActionFilesInput,
+ "waitdom": ActionWaitDOM,
+ "waitfcp": ActionWaitFCP,
+ "waitfmp": ActionWaitFMP,
+ "waitidle": ActionWaitIdle,
"waitload": ActionWaitLoad,
+ "waitstable": ActionWaitStable,
"getresource": ActionGetResource,
"extract": ActionExtract,
"setmethod": ActionSetMethod,
@@ -104,6 +131,7 @@ var ActionStringToAction = map[string]ActionType{
"deleteheader": ActionDeleteHeader,
"setbody": ActionSetBody,
"waitevent": ActionWaitEvent,
+ "waitdialog": ActionWaitDialog,
"keyboard": ActionKeyboard,
"debug": ActionDebug,
"sleep": ActionSleep,
@@ -121,7 +149,12 @@ var ActionToActionString = map[ActionType]string{
ActionTimeInput: "time",
ActionSelectInput: "select",
ActionFilesInput: "files",
+ ActionWaitDOM: "waitdom",
+ ActionWaitFCP: "waitfcp",
+ ActionWaitFMP: "waitfmp",
+ ActionWaitIdle: "waitidle",
ActionWaitLoad: "waitload",
+ ActionWaitStable: "waitstable",
ActionGetResource: "getresource",
ActionExtract: "extract",
ActionSetMethod: "setmethod",
@@ -130,6 +163,7 @@ var ActionToActionString = map[ActionType]string{
ActionDeleteHeader: "deleteheader",
ActionSetBody: "setbody",
ActionWaitEvent: "waitevent",
+ ActionWaitDialog: "waitdialog",
ActionKeyboard: "keyboard",
ActionDebug: "debug",
ActionSleep: "sleep",
diff --git a/pkg/protocols/headless/engine/engine.go b/pkg/protocols/headless/engine/engine.go
index 94c2928569..308fced49f 100644
--- a/pkg/protocols/headless/engine/engine.go
+++ b/pkg/protocols/headless/engine/engine.go
@@ -11,7 +11,15 @@ import (
"github.com/go-rod/rod/lib/launcher/flags"
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/headless/engine/engine.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/headless/engine/engine.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
osutils "github.com/projectdiscovery/utils/os"
processutil "github.com/projectdiscovery/utils/process"
diff --git a/pkg/protocols/headless/engine/http_client.go b/pkg/protocols/headless/engine/http_client.go
index cd9476ffd8..2f98a0f3a5 100644
--- a/pkg/protocols/headless/engine/http_client.go
+++ b/pkg/protocols/headless/engine/http_client.go
@@ -15,6 +15,19 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/utils"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/fastdialer/fastdialer/ja3/impersonate"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/headless/engine/http_client.go
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/headless/engine/http_client.go
+>>>>>>> projectdiscovery-main
)
// newHttpClient creates a new http client for headless communication with a timeout
diff --git a/pkg/protocols/headless/engine/instance.go b/pkg/protocols/headless/engine/instance.go
index 8e9b5d6818..44a62aff80 100644
--- a/pkg/protocols/headless/engine/instance.go
+++ b/pkg/protocols/headless/engine/instance.go
@@ -8,6 +8,14 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/interactsh"
"github.com/go-rod/rod"
"github.com/go-rod/rod/lib/utils"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/headless/engine/instance.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/interactsh"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/headless/engine/instance.go
+>>>>>>> projectdiscovery-main
)
// Instance is an isolated browser instance opened for doing operations with it.
diff --git a/pkg/protocols/headless/engine/page.go b/pkg/protocols/headless/engine/page.go
index a88f16cf0f..46d9298b7d 100644
--- a/pkg/protocols/headless/engine/page.go
+++ b/pkg/protocols/headless/engine/page.go
@@ -45,7 +45,7 @@ type Options struct {
}
// Run runs a list of actions by creating a new page in the browser.
-func (i *Instance) Run(input *contextargs.Context, actions []*Action, payloads map[string]interface{}, options *Options) (map[string]string, *Page, error) {
+func (i *Instance) Run(input *contextargs.Context, actions []*Action, payloads map[string]interface{}, options *Options) (ActionData, *Page, error) {
page, err := i.engine.Page(proto.TargetCreateTarget{})
if err != nil {
return nil, nil, err
diff --git a/pkg/protocols/headless/engine/page_actions.go b/pkg/protocols/headless/engine/page_actions.go
index ce68c896ad..602d4a7d11 100644
--- a/pkg/protocols/headless/engine/page_actions.go
+++ b/pkg/protocols/headless/engine/page_actions.go
@@ -21,9 +21,31 @@ import (
"github.com/go-rod/rod/lib/input"
"github.com/go-rod/rod/lib/proto"
"github.com/go-rod/rod/lib/utils"
+ "github.com/kitabisa/go-ci"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
contextutil "github.com/projectdiscovery/utils/context"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/headless/engine/page_actions.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ httputil "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils/http"
+ contextutil "github.com/projectdiscovery/utils/context"
+ "github.com/projectdiscovery/utils/errkit"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/headless/engine/page_actions.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
folderutil "github.com/projectdiscovery/utils/folder"
@@ -35,6 +57,8 @@ import (
var (
errinvalidArguments = errorutil.New("invalid arguments provided")
ErrLFAccessDenied = errorutil.New("Use -allow-local-file-access flag to enable local file access")
+ // ErrActionExecDealine is the error returned when alloted time for action execution exceeds
+ ErrActionExecDealine = errkit.New("headless action execution deadline exceeded").SetKind(errkit.ErrKindDeadline).Build()
)
const (
@@ -44,14 +68,19 @@ const (
)
// ExecuteActions executes a list of actions on a page.
-func (p *Page) ExecuteActions(input *contextargs.Context, actions []*Action, variables map[string]interface{}) (outData map[string]string, err error) {
- outData = make(map[string]string)
+func (p *Page) ExecuteActions(input *contextargs.Context, actions []*Action, variables map[string]interface{}) (outData ActionData, err error) {
+ outData = make(ActionData)
// waitFuncs are function that needs to be executed after navigation
// typically used for waitEvent
waitFuncs := make([]func() error, 0)
// avoid any future panics caused due to go-rod library
+ // TODO(dwisiswant0): remove this once we get the RCA.
defer func() {
+ if ci.IsCI() {
+ return
+ }
+
if r := recover(); r != nil {
err = errorutil.New("panic on headless action: %v", r)
}
@@ -85,8 +114,28 @@ func (p *Page) ExecuteActions(input *contextargs.Context, actions []*Action, var
err = p.TimeInputElement(act, outData)
case ActionSelectInput:
err = p.SelectInputElement(act, outData)
+ case ActionWaitDOM:
+ event := proto.PageLifecycleEventNameDOMContentLoaded
+ err = p.WaitPageLifecycleEvent(act, outData, event)
+ case ActionWaitFCP:
+ event := proto.PageLifecycleEventNameFirstContentfulPaint
+ err = p.WaitPageLifecycleEvent(act, outData, event)
+ case ActionWaitFMP:
+ event := proto.PageLifecycleEventNameFirstMeaningfulPaint
+ err = p.WaitPageLifecycleEvent(act, outData, event)
+ case ActionWaitIdle:
+ event := proto.PageLifecycleEventNameNetworkIdle
+ err = p.WaitPageLifecycleEvent(act, outData, event)
case ActionWaitLoad:
- err = p.WaitLoad(act, outData)
+ event := proto.PageLifecycleEventNameLoad
+ err = p.WaitPageLifecycleEvent(act, outData, event)
+ case ActionWaitStable:
+ err = p.WaitStable(act, outData)
+ // NOTE(dwisiswant0): Mapping `ActionWaitLoad` to `Page.WaitStable`,
+ // just in case waiting for the `proto.PageLifecycleEventNameLoad` event
+ // doesn't meet expectations.
+ // case ActionWaitLoad, ActionWaitStable:
+ // err = p.WaitStable(act, outData)
case ActionGetResource:
err = p.GetResource(act, outData)
case ActionExtract:
@@ -97,6 +146,8 @@ func (p *Page) ExecuteActions(input *contextargs.Context, actions []*Action, var
if waitFunc != nil {
waitFuncs = append(waitFuncs, waitFunc)
}
+ case ActionWaitDialog:
+ err = p.HandleDialog(act, outData)
case ActionFilesInput:
if p.options.Options.AllowLocalFileAccess {
err = p.FilesInput(act, outData)
@@ -139,7 +190,7 @@ type rule struct {
}
// WaitVisible waits until an element appears.
-func (p *Page) WaitVisible(act *Action, out map[string]string) error {
+func (p *Page) WaitVisible(act *Action, out ActionData) error {
timeout, err := getTimeout(p, act)
if err != nil {
return errors.Wrap(err, "Wrong timeout given")
@@ -193,6 +244,17 @@ func createBackOffSleeper(pollTimeout, timeout time.Duration) utils.Sleeper {
}
}
+func getNavigationFunc(p *Page, act *Action, event proto.PageLifecycleEventName) (func(), error) {
+ dur, err := getTimeout(p, act)
+ if err != nil {
+ return nil, errors.Wrap(err, "Wrong timeout given")
+ }
+
+ fn := p.page.Timeout(dur).WaitNavigation(event)
+
+ return fn, nil
+}
+
func getTimeout(p *Page, act *Action) (time.Duration, error) {
return geTimeParameter(p, act, "timeout", 3, time.Second)
}
@@ -214,7 +276,7 @@ func geTimeParameter(p *Page, act *Action, parameterName string, defaultValue ti
}
// ActionAddHeader executes a AddHeader action.
-func (p *Page) ActionAddHeader(act *Action, out map[string]string) error {
+func (p *Page) ActionAddHeader(act *Action, out ActionData) error {
in := p.getActionArgWithDefaultValues(act, "part")
args := make(map[string]string)
@@ -225,7 +287,7 @@ func (p *Page) ActionAddHeader(act *Action, out map[string]string) error {
}
// ActionSetHeader executes a SetHeader action.
-func (p *Page) ActionSetHeader(act *Action, out map[string]string) error {
+func (p *Page) ActionSetHeader(act *Action, out ActionData) error {
in := p.getActionArgWithDefaultValues(act, "part")
args := make(map[string]string)
@@ -236,7 +298,7 @@ func (p *Page) ActionSetHeader(act *Action, out map[string]string) error {
}
// ActionDeleteHeader executes a DeleteHeader action.
-func (p *Page) ActionDeleteHeader(act *Action, out map[string]string) error {
+func (p *Page) ActionDeleteHeader(act *Action, out ActionData) error {
in := p.getActionArgWithDefaultValues(act, "part")
args := make(map[string]string)
@@ -246,7 +308,7 @@ func (p *Page) ActionDeleteHeader(act *Action, out map[string]string) error {
}
// ActionSetBody executes a SetBody action.
-func (p *Page) ActionSetBody(act *Action, out map[string]string) error {
+func (p *Page) ActionSetBody(act *Action, out ActionData) error {
in := p.getActionArgWithDefaultValues(act, "part")
args := make(map[string]string)
@@ -256,7 +318,7 @@ func (p *Page) ActionSetBody(act *Action, out map[string]string) error {
}
// ActionSetMethod executes an SetMethod action.
-func (p *Page) ActionSetMethod(act *Action, out map[string]string) error {
+func (p *Page) ActionSetMethod(act *Action, out ActionData) error {
in := p.getActionArgWithDefaultValues(act, "part")
args := make(map[string]string)
@@ -266,7 +328,7 @@ func (p *Page) ActionSetMethod(act *Action, out map[string]string) error {
}
// NavigateURL executes an ActionLoadURL actions loading a URL for the page.
-func (p *Page) NavigateURL(action *Action, out map[string]string, allvars map[string]interface{}) error {
+func (p *Page) NavigateURL(action *Action, out ActionData, allvars map[string]interface{}) error {
// input <- is input url from cli
// target <- is the url from template (ex: {{BaseURL}}/test)
input, err := urlutil.Parse(p.input.MetaInput.Input)
@@ -292,7 +354,7 @@ func (p *Page) NavigateURL(action *Action, out map[string]string, allvars map[st
allvars = generators.MergeMaps(allvars, defaultReqVars)
if vardump.EnableVarDump {
- gologger.Debug().Msgf("Headless Protocol request variables: \n%s\n", vardump.DumpVariables(allvars))
+ gologger.Debug().Msgf("Headless Protocol request variables: %s\n", vardump.DumpVariables(allvars))
}
// Evaluate the target url with all variables
@@ -322,7 +384,7 @@ func (p *Page) NavigateURL(action *Action, out map[string]string, allvars map[st
}
// RunScript runs a script on the loaded page
-func (p *Page) RunScript(action *Action, out map[string]string) error {
+func (p *Page) RunScript(action *Action, out ActionData) error {
code := p.getActionArgWithDefaultValues(action, "code")
if code == "" {
return errinvalidArguments
@@ -343,7 +405,7 @@ func (p *Page) RunScript(action *Action, out map[string]string) error {
}
// ClickElement executes click actions for an element.
-func (p *Page) ClickElement(act *Action, out map[string]string) error {
+func (p *Page) ClickElement(act *Action, out ActionData) error {
element, err := p.pageElementBy(act.Data)
if err != nil {
return errors.Wrap(err, errCouldNotGetElement)
@@ -358,12 +420,12 @@ func (p *Page) ClickElement(act *Action, out map[string]string) error {
}
// KeyboardAction executes a keyboard action on the page.
-func (p *Page) KeyboardAction(act *Action, out map[string]string) error {
+func (p *Page) KeyboardAction(act *Action, out ActionData) error {
return p.page.Keyboard.Type([]input.Key(p.getActionArgWithDefaultValues(act, "keys"))...)
}
// RightClickElement executes right click actions for an element.
-func (p *Page) RightClickElement(act *Action, out map[string]string) error {
+func (p *Page) RightClickElement(act *Action, out ActionData) error {
element, err := p.pageElementBy(act.Data)
if err != nil {
return errors.Wrap(err, errCouldNotGetElement)
@@ -378,7 +440,7 @@ func (p *Page) RightClickElement(act *Action, out map[string]string) error {
}
// Screenshot executes screenshot action on a page
-func (p *Page) Screenshot(act *Action, out map[string]string) error {
+func (p *Page) Screenshot(act *Action, out ActionData) error {
to := p.getActionArgWithDefaultValues(act, "to")
if to == "" {
to = ksuid.New().String()
@@ -441,7 +503,7 @@ func (p *Page) Screenshot(act *Action, out map[string]string) error {
}
// InputElement executes input element actions for an element.
-func (p *Page) InputElement(act *Action, out map[string]string) error {
+func (p *Page) InputElement(act *Action, out ActionData) error {
value := p.getActionArgWithDefaultValues(act, "value")
if value == "" {
return errinvalidArguments
@@ -460,7 +522,7 @@ func (p *Page) InputElement(act *Action, out map[string]string) error {
}
// TimeInputElement executes time input on an element
-func (p *Page) TimeInputElement(act *Action, out map[string]string) error {
+func (p *Page) TimeInputElement(act *Action, out ActionData) error {
value := p.getActionArgWithDefaultValues(act, "value")
if value == "" {
return errinvalidArguments
@@ -483,7 +545,7 @@ func (p *Page) TimeInputElement(act *Action, out map[string]string) error {
}
// SelectInputElement executes select input statement action on a element
-func (p *Page) SelectInputElement(act *Action, out map[string]string) error {
+func (p *Page) SelectInputElement(act *Action, out ActionData) error {
value := p.getActionArgWithDefaultValues(act, "value")
if value == "" {
return errinvalidArguments
@@ -507,22 +569,40 @@ func (p *Page) SelectInputElement(act *Action, out map[string]string) error {
return nil
}
-// WaitLoad waits for the page to load
-func (p *Page) WaitLoad(act *Action, out map[string]string) error {
- p.page.Timeout(2 * time.Second).WaitNavigation(proto.PageLifecycleEventNameFirstMeaningfulPaint)()
-
- // Wait for the window.onload event and also wait for the network requests
- // to become idle for a maximum duration of 3 seconds. If the requests
- // do not finish,
- if err := p.page.WaitLoad(); err != nil {
- return errors.Wrap(err, "could not wait load event")
+// WaitPageLifecycleEvent waits for specified page lifecycle event name
+func (p *Page) WaitPageLifecycleEvent(act *Action, out ActionData, event proto.PageLifecycleEventName) error {
+ fn, err := getNavigationFunc(p, act, event)
+ if err != nil {
+ return err
}
- _ = p.page.WaitIdle(1 * time.Second)
+
+ fn()
+
return nil
}
+// WaitStable waits until the page is stable
+func (p *Page) WaitStable(act *Action, out ActionData) error {
+ var dur time.Duration = time.Second // default stable page duration: 1s
+
+ timeout, err := getTimeout(p, act)
+ if err != nil {
+ return errors.Wrap(err, "Wrong timeout given")
+ }
+
+ argDur := act.Data["duration"]
+ if argDur != "" {
+ dur, err = time.ParseDuration(argDur)
+ if err != nil {
+ dur = time.Second
+ }
+ }
+
+ return p.page.Timeout(timeout).WaitStable(dur)
+}
+
// GetResource gets a resource from an element from page.
-func (p *Page) GetResource(act *Action, out map[string]string) error {
+func (p *Page) GetResource(act *Action, out ActionData) error {
element, err := p.pageElementBy(act.Data)
if err != nil {
return errors.Wrap(err, errCouldNotGetElement)
@@ -538,7 +618,7 @@ func (p *Page) GetResource(act *Action, out map[string]string) error {
}
// FilesInput acts with a file input element on page
-func (p *Page) FilesInput(act *Action, out map[string]string) error {
+func (p *Page) FilesInput(act *Action, out ActionData) error {
element, err := p.pageElementBy(act.Data)
if err != nil {
return errors.Wrap(err, errCouldNotGetElement)
@@ -555,7 +635,7 @@ func (p *Page) FilesInput(act *Action, out map[string]string) error {
}
// ExtractElement extracts from an element on the page.
-func (p *Page) ExtractElement(act *Action, out map[string]string) error {
+func (p *Page) ExtractElement(act *Action, out ActionData) error {
element, err := p.pageElementBy(act.Data)
if err != nil {
return errors.Wrap(err, errCouldNotGetElement)
@@ -589,7 +669,7 @@ func (p *Page) ExtractElement(act *Action, out map[string]string) error {
}
// WaitEvent waits for an event to happen on the page.
-func (p *Page) WaitEvent(act *Action, out map[string]string) (func() error, error) {
+func (p *Page) WaitEvent(act *Action, out ActionData) (func() error, error) {
event := p.getActionArgWithDefaultValues(act, "event")
if event == "" {
return nil, errors.New("event not recognized")
@@ -619,7 +699,7 @@ func (p *Page) WaitEvent(act *Action, out map[string]string) (func() error, erro
// Just wait the event to happen
waitFunc := func() (err error) {
// execute actual wait event
- ctx, cancel := context.WithTimeout(context.Background(), maxDuration)
+ ctx, cancel := context.WithTimeoutCause(context.Background(), maxDuration, ErrActionExecDealine)
defer cancel()
err = contextutil.ExecFunc(ctx, p.page.WaitEvent(waitEvent))
return
@@ -627,6 +707,43 @@ func (p *Page) WaitEvent(act *Action, out map[string]string) (func() error, erro
return waitFunc, nil
}
+// HandleDialog handles JavaScript dialog (alert, confirm, prompt, or onbeforeunload).
+func (p *Page) HandleDialog(act *Action, out ActionData) error {
+ maxDuration := 10 * time.Second
+
+ if dur := p.getActionArgWithDefaultValues(act, "max-duration"); dur != "" {
+ var err error
+
+ maxDuration, err = time.ParseDuration(dur)
+ if err != nil {
+ return errorutil.NewWithErr(err).Msgf("could not parse max-duration")
+ }
+ }
+
+ ctx, cancel := context.WithTimeout(context.Background(), maxDuration)
+ defer cancel()
+
+ wait, handle := p.page.HandleDialog()
+ fn := func() (*proto.PageJavascriptDialogOpening, error) {
+ dialog := wait()
+ err := handle(&proto.PageHandleJavaScriptDialog{
+ Accept: true,
+ PromptText: "",
+ })
+
+ return dialog, err
+ }
+
+ dialog, err := contextutil.ExecFuncWithTwoReturns(ctx, fn)
+ if err == nil && act.Name != "" {
+ out[act.Name] = true
+ out[act.Name+"_type"] = string(dialog.Type)
+ out[act.Name+"_message"] = dialog.Message
+ }
+
+ return nil
+}
+
// pageElementBy returns a page element from a variety of inputs.
//
// Supported values for by: r -> selector & regex, x -> xpath, js -> eval js,
@@ -661,14 +778,14 @@ func (p *Page) pageElementBy(data map[string]string) (*rod.Element, error) {
}
// DebugAction enables debug action on a page.
-func (p *Page) DebugAction(act *Action, out map[string]string) error {
+func (p *Page) DebugAction(act *Action, out ActionData) error {
p.instance.browser.engine.SlowMotion(5 * time.Second)
p.instance.browser.engine.Trace(true)
return nil
}
// SleepAction sleeps on the page for a specified duration
-func (p *Page) SleepAction(act *Action, out map[string]string) error {
+func (p *Page) SleepAction(act *Action, out ActionData) error {
seconds := act.Data["duration"]
if seconds == "" {
seconds = "5"
diff --git a/pkg/protocols/headless/engine/page_actions_test.go b/pkg/protocols/headless/engine/page_actions_test.go
index c2fa66f461..368446e209 100644
--- a/pkg/protocols/headless/engine/page_actions_test.go
+++ b/pkg/protocols/headless/engine/page_actions_test.go
@@ -18,11 +18,32 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/Explorer1092/nuclei/v3/pkg/testutils/testheadless"
"github.com/Explorer1092/nuclei/v3/pkg/types"
stringsutil "github.com/projectdiscovery/utils/strings"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/headless/engine/page_actions_test.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolstate"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils/testheadless"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v2/pkg/testutils/testheadless"
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils/testheadless"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ stringsutil "github.com/projectdiscovery/utils/strings"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/headless/engine/page_actions_test.go
+>>>>>>> projectdiscovery-main
)
func TestActionNavigate(t *testing.T) {
@@ -38,7 +59,7 @@ func TestActionNavigate(t *testing.T) {
actions := []*Action{{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}}, {ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}}}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nilf(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
})
@@ -63,7 +84,7 @@ func TestActionScript(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionScript}, Name: "test", Data: map[string]string{"code": "() => window.test"}},
}
- testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
require.Equal(t, "some-data", out["test"], "could not run js and get results correctly")
@@ -77,7 +98,7 @@ func TestActionScript(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionScript}, Name: "test", Data: map[string]string{"code": "() => window.test"}},
}
- testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
require.Equal(t, "some-data", out["test"], "could not run js and get results correctly with js hook")
@@ -101,7 +122,7 @@ func TestActionClick(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionClick}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
el := page.Page().MustElement("button")
@@ -134,7 +155,7 @@ func TestActionRightClick(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionRightClick}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
el := page.Page().MustElement("button")
@@ -159,7 +180,7 @@ func TestActionTextInput(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionTextInput}, Data: map[string]string{"selector": "input", "value": "test"}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
el := page.Page().MustElement("input")
@@ -182,7 +203,7 @@ func TestActionHeadersChange(t *testing.T) {
}
}
- testHeadless(t, actions, 20*time.Second, handler, func(page *Page, err error, out map[string]string) {
+ testHeadless(t, actions, 20*time.Second, handler, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "found", strings.ToLower(strings.TrimSpace(page.Page().MustElement("html").MustText())), "could not set header correctly")
})
@@ -201,11 +222,11 @@ func TestActionScreenshot(t *testing.T) {
filePath := filepath.Join(os.TempDir(), "test.png")
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
- {ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
+ {ActionType: ActionTypeHolder{ActionType: ActionWaitFMP}},
{ActionType: ActionTypeHolder{ActionType: ActionScreenshot}, Data: map[string]string{"to": filePath}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
_ = page.Page()
@@ -229,11 +250,11 @@ func TestActionScreenshotToDir(t *testing.T) {
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
- {ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
+ {ActionType: ActionTypeHolder{ActionType: ActionWaitFMP}},
{ActionType: ActionTypeHolder{ActionType: ActionScreenshot}, Data: map[string]string{"to": filePath, "mkdir": "true"}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
_ = page.Page()
@@ -260,7 +281,7 @@ func TestActionTimeInput(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionTimeInput}, Data: map[string]string{"selector": "input", "value": "2006-01-02T15:04:05Z"}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
el := page.Page().MustElement("input")
@@ -288,7 +309,7 @@ func TestActionSelectInput(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionSelectInput}, Data: map[string]string{"by": "x", "xpath": "//select[@id='test']", "value": "Test2", "selected": "true"}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
el := page.Page().MustElement("select")
require.Equal(t, "Test2", el.MustText(), "could not get input change value")
@@ -311,7 +332,7 @@ func TestActionFilesInput(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionFilesInput}, Data: map[string]string{"selector": "input", "value": "test1.pdf"}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Nuclei Test Page", page.Page().MustInfo().Title, "could not navigate correctly")
el := page.Page().MustElement("input")
@@ -337,7 +358,7 @@ func TestActionFilesInputNegative(t *testing.T) {
}
t.Setenv("LOCAL_FILE_ACCESS", "false")
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.ErrorContains(t, err, ErrLFAccessDenied.Error(), "got file access when -lfa is false")
})
}
@@ -359,7 +380,7 @@ func TestActionWaitLoad(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
el := page.Page().MustElement("button")
style, attributeErr := el.Attribute("style")
@@ -384,9 +405,12 @@ func TestActionGetResource(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionGetResource}, Data: map[string]string{"by": "x", "xpath": "//img[@id='test']"}, Name: "src"},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
- require.Equal(t, len(out["src"]), 121808, "could not find resource")
+
+ src, ok := out["src"].(string)
+ require.True(t, ok, "could not assert src to string")
+ require.Equal(t, len(src), 121808, "could not find resource")
})
}
@@ -404,7 +428,7 @@ func TestActionExtract(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionExtract}, Data: map[string]string{"by": "x", "xpath": "//button[@id='test']"}, Name: "extract"},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "Wait for me!", out["extract"], "could not extract text")
})
@@ -423,7 +447,7 @@ func TestActionSetMethod(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionSetMethod}, Data: map[string]string{"part": "x", "method": "SET"}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "SET", page.rules[0].Args["method"], "could not find resource")
})
@@ -442,7 +466,7 @@ func TestActionAddHeader(t *testing.T) {
}
}
- testHeadless(t, actions, 20*time.Second, handler, func(page *Page, err error, out map[string]string) {
+ testHeadless(t, actions, 20*time.Second, handler, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "found", strings.ToLower(strings.TrimSpace(page.Page().MustElement("html").MustText())), "could not set header correctly")
})
@@ -463,7 +487,7 @@ func TestActionDeleteHeader(t *testing.T) {
}
}
- testHeadless(t, actions, 20*time.Second, handler, func(page *Page, err error, out map[string]string) {
+ testHeadless(t, actions, 20*time.Second, handler, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "header deleted", strings.ToLower(strings.TrimSpace(page.Page().MustElement("html").MustText())), "could not delete header correctly")
})
@@ -481,7 +505,7 @@ func TestActionSetBody(t *testing.T) {
_, _ = fmt.Fprintln(w, string(body))
}
- testHeadless(t, actions, 20*time.Second, handler, func(page *Page, err error, out map[string]string) {
+ testHeadless(t, actions, 20*time.Second, handler, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.Equal(t, "hello", strings.ToLower(strings.TrimSpace(page.Page().MustElement("html").MustText())), "could not set header correctly")
})
@@ -505,7 +529,7 @@ func TestActionKeyboard(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionKeyboard}, Data: map[string]string{"keys": "Test2"}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
el := page.Page().MustElement("input")
require.Equal(t, "Test2", el.MustText(), "could not get input change value")
@@ -529,7 +553,7 @@ func TestActionSleep(t *testing.T) {
{ActionType: ActionTypeHolder{ActionType: ActionSleep}, Data: map[string]string{"duration": "2"}},
}
- testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
require.True(t, page.Page().MustElement("button").MustVisible(), "could not get button")
})
@@ -553,7 +577,7 @@ func TestActionWaitVisible(t *testing.T) {
}
t.Run("wait for an element being visible", func(t *testing.T) {
- testHeadlessSimpleResponse(t, response, actions, 2*time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, 2*time.Second, func(page *Page, err error, out ActionData) {
require.Nil(t, err, "could not run page actions")
page.Page().MustElement("button").MustVisible()
@@ -562,21 +586,82 @@ func TestActionWaitVisible(t *testing.T) {
t.Run("timeout because of element not visible", func(t *testing.T) {
// increased timeout from time.Second/2 to time.Second due to random fails (probably due to overhead and system)
- testHeadlessSimpleResponse(t, response, actions, time.Second, func(page *Page, err error, out map[string]string) {
+ testHeadlessSimpleResponse(t, response, actions, time.Second, func(page *Page, err error, out ActionData) {
require.Error(t, err)
require.Contains(t, err.Error(), "Element did not appear in the given amount of time")
})
})
}
-func testHeadlessSimpleResponse(t *testing.T, response string, actions []*Action, timeout time.Duration, assert func(page *Page, pageErr error, out map[string]string)) {
+func TestActionWaitDialog(t *testing.T) {
+ response := `
+
+ Nuclei Test Page
+
+
+
+
+ `
+
+ t.Run("Triggered", func(t *testing.T) {
+ actions := []*Action{
+ {
+ ActionType: ActionTypeHolder{ActionType: ActionNavigate},
+ Data: map[string]string{"url": "{{BaseURL}}/?script=alert%281%29"},
+ },
+ {
+ ActionType: ActionTypeHolder{ActionType: ActionWaitDialog},
+ Name: "test",
+ },
+ }
+
+ testHeadlessSimpleResponse(t, response, actions, 1*time.Second, func(page *Page, err error, out ActionData) {
+ require.Nil(t, err, "could not run page actions")
+
+ test, ok := out["test"].(bool)
+ require.True(t, ok, "could not assert test to bool")
+ require.True(t, test, "could not find test")
+ })
+ })
+
+ t.Run("Invalid", func(t *testing.T) {
+ actions := []*Action{
+ {
+ ActionType: ActionTypeHolder{ActionType: ActionNavigate},
+ Data: map[string]string{"url": "{{BaseURL}}/?script=foo"},
+ },
+ {
+ ActionType: ActionTypeHolder{ActionType: ActionWaitDialog},
+ Name: "test",
+ },
+ }
+
+ testHeadlessSimpleResponse(t, response, actions, 1*time.Second, func(page *Page, err error, out ActionData) {
+ require.Nil(t, err, "could not run page actions")
+
+ _, ok := out["test"].(bool)
+ require.False(t, ok, "output assertion is success")
+ })
+ })
+}
+
+func testHeadlessSimpleResponse(t *testing.T, response string, actions []*Action, timeout time.Duration, assert func(page *Page, pageErr error, out ActionData)) {
t.Helper()
testHeadless(t, actions, timeout, func(w http.ResponseWriter, r *http.Request) {
_, _ = fmt.Fprintln(w, response)
}, assert)
}
-func testHeadless(t *testing.T, actions []*Action, timeout time.Duration, handler func(w http.ResponseWriter, r *http.Request), assert func(page *Page, pageErr error, extractedData map[string]string)) {
+func testHeadless(t *testing.T, actions []*Action, timeout time.Duration, handler func(w http.ResponseWriter, r *http.Request), assert func(page *Page, pageErr error, extractedData ActionData)) {
t.Helper()
lfa := getBoolFromEnv("LOCAL_FILE_ACCESS", true)
diff --git a/pkg/protocols/headless/engine/util.go b/pkg/protocols/headless/engine/util.go
index 25e64328e6..6374e67ac9 100644
--- a/pkg/protocols/headless/engine/util.go
+++ b/pkg/protocols/headless/engine/util.go
@@ -1,7 +1,15 @@
package engine
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/marker"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/headless/engine/util.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/marker"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/marker"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/headless/engine/util.go
+>>>>>>> projectdiscovery-main
"github.com/valyala/fasttemplate"
)
diff --git a/pkg/protocols/headless/headless.go b/pkg/protocols/headless/headless.go
index 755c416dc7..90ebaab29a 100644
--- a/pkg/protocols/headless/headless.go
+++ b/pkg/protocols/headless/headless.go
@@ -3,6 +3,7 @@ package headless
import (
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/fuzz"
useragent "github.com/Explorer1092/nuclei/v3/pkg/model/types/userAgent"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
@@ -10,6 +11,32 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/headless/engine"
uagent "github.com/projectdiscovery/useragent"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/headless/headless.go
+<<<<<<< HEAD
+ useragent "github.com/Explorer1092/nuclei/v2/pkg/model/types/userAgent"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/headless/engine"
+=======
+ useragent "github.com/projectdiscovery/nuclei/v2/pkg/model/types/userAgent"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/fuzz"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/headless/engine"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz"
+ useragent "github.com/projectdiscovery/nuclei/v3/pkg/model/types/userAgent"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless/engine"
+ uagent "github.com/projectdiscovery/useragent"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/headless/headless.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
)
diff --git a/pkg/protocols/headless/operators.go b/pkg/protocols/headless/operators.go
index 7104ba3e28..936432556a 100644
--- a/pkg/protocols/headless/operators.go
+++ b/pkg/protocols/headless/operators.go
@@ -4,6 +4,7 @@ import (
"strconv"
"time"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/operators/extractors"
@@ -12,6 +13,26 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
protocolUtils "github.com/Explorer1092/nuclei/v3/pkg/protocols/utils"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/headless/operators.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ protocolUtils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/headless/operators.go
+>>>>>>> projectdiscovery-main
)
// Match matches a generic data response again a given matcher
@@ -137,6 +158,7 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(wrapped.InternalEvent["template-id"]),
TemplatePath: types.ToString(wrapped.InternalEvent["template-path"]),
Info: wrapped.InternalEvent["template-info"].(model.Info),
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Host: fields.Host,
Path: fields.Path,
diff --git a/pkg/protocols/headless/request.go b/pkg/protocols/headless/request.go
index 7212721405..7957db61ee 100644
--- a/pkg/protocols/headless/request.go
+++ b/pkg/protocols/headless/request.go
@@ -25,12 +25,55 @@ import (
templateTypes "github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/headless/request.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/interactsh"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http/utils"
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/output"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/fuzz"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/headless/engine"
+ protocolutils "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless/engine"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/headless/request.go
+>>>>>>> projectdiscovery-main
urlutil "github.com/projectdiscovery/utils/url"
)
var _ protocols.Request = &Request{}
-const errCouldGetHtmlElement = "could get html element"
+const errCouldNotGetHtmlElement = "could not get html element"
// Type returns the type of the protocol request
func (request *Request) Type() templateTypes.ProtocolType {
@@ -117,12 +160,12 @@ func (request *Request) executeRequestWithPayloads(input *contextargs.Context, p
if err != nil {
request.options.Output.Request(request.options.TemplatePath, input.MetaInput.Input, request.Type().String(), err)
request.options.Progress.IncrementFailedRequestsBy(1)
- return errors.Wrap(err, errCouldGetHtmlElement)
+ return errors.Wrap(err, errCouldNotGetHtmlElement)
}
defer instance.Close()
if vardump.EnableVarDump {
- gologger.Debug().Msgf("Headless Protocol request variables: \n%s\n", vardump.DumpVariables(payloads))
+ gologger.Debug().Msgf("Headless Protocol request variables: %s\n", vardump.DumpVariables(payloads))
}
instance.SetInteractsh(request.options.Interactsh)
@@ -130,7 +173,7 @@ func (request *Request) executeRequestWithPayloads(input *contextargs.Context, p
if _, err := url.Parse(input.MetaInput.Input); err != nil {
request.options.Output.Request(request.options.TemplatePath, input.MetaInput.Input, request.Type().String(), err)
request.options.Progress.IncrementFailedRequestsBy(1)
- return errors.Wrap(err, errCouldGetHtmlElement)
+ return errors.Wrap(err, errCouldNotGetHtmlElement)
}
options := &engine.Options{
Timeout: time.Duration(request.options.Options.PageTimeout) * time.Second,
@@ -146,7 +189,7 @@ func (request *Request) executeRequestWithPayloads(input *contextargs.Context, p
if err != nil {
request.options.Output.Request(request.options.TemplatePath, input.MetaInput.Input, request.Type().String(), err)
request.options.Progress.IncrementFailedRequestsBy(1)
- return errors.Wrap(err, errCouldGetHtmlElement)
+ return errors.Wrap(err, errCouldNotGetHtmlElement)
}
defer page.Close()
@@ -174,7 +217,7 @@ func (request *Request) executeRequestWithPayloads(input *contextargs.Context, p
reqBuilder.WriteString("\t" + actStepStr + "\n")
}
}
- gologger.Debug().Msgf(reqBuilder.String())
+ gologger.Debug().Msg(reqBuilder.String())
}
var responseBody string
@@ -183,7 +226,13 @@ func (request *Request) executeRequestWithPayloads(input *contextargs.Context, p
responseBody, _ = html.HTML()
}
- outputEvent := request.responseToDSLMap(responseBody, out["header"], out["status_code"], reqBuilder.String(), input.MetaInput.Input, navigatedURL, page.DumpHistory())
+ header := out.GetOrDefault("header", "").(string)
+
+ // NOTE(dwisiswant0): `status_code` key should be an integer type.
+ // Ref: https://github.com/projectdiscovery/nuclei/pull/5545#discussion_r1721291013
+ statusCode := out.GetOrDefault("status_code", "").(string)
+
+ outputEvent := request.responseToDSLMap(responseBody, header, statusCode, reqBuilder.String(), input.MetaInput.Input, navigatedURL, page.DumpHistory())
// add response fields to template context and merge templatectx variables to output event
request.options.AddTemplateVars(input.MetaInput, request.Type(), request.ID, outputEvent)
if request.options.HasTemplateCtx(input.MetaInput) {
@@ -215,6 +264,10 @@ func (request *Request) executeRequestWithPayloads(input *contextargs.Context, p
}
dumpResponse(event, request.options, responseBody, input.MetaInput.Input)
+ shouldStopAtFirstMatch := request.StopAtFirstMatch || request.options.StopAtFirstMatch || request.options.Options.StopAtFirstMatch
+ if shouldStopAtFirstMatch && event.HasOperatorResult() {
+ return types.ErrNoMoreRequests
+ }
return nil
}
diff --git a/pkg/protocols/http/build_request.go b/pkg/protocols/http/build_request.go
index 7fcf2bf937..7b6d043a22 100644
--- a/pkg/protocols/http/build_request.go
+++ b/pkg/protocols/http/build_request.go
@@ -24,6 +24,45 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/Explorer1092/nuclei/v3/pkg/types/scanstrategy"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/build_request.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/replacer"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http/race"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http/raw"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http/utils"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/race"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/raw"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/utils"
+ protocolutils "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/race"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/raw"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ httputil "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils/http"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types/scanstrategy"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/build_request.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/rawhttp"
"github.com/projectdiscovery/retryablehttp-go"
errorutil "github.com/projectdiscovery/utils/errors"
@@ -55,6 +94,8 @@ type generatedRequest struct {
// requestURLPattern tracks unmodified request url pattern without values ( it is used for constant vuln_hash)
// ex: {{BaseURL}}/api/exp?param={{randstr}}
requestURLPattern string
+
+ fuzzGeneratedRequest fuzz.GeneratedRequest
}
// setReqURLPattern sets the url request pattern for the generated request
@@ -90,9 +131,9 @@ func (g *generatedRequest) ApplyAuth(provider authprovider.AuthProvider) {
return
}
if g.request != nil {
- auth := provider.LookupURLX(g.request.URL)
- if auth != nil {
- auth.ApplyOnRR(g.request)
+ authStrategies := provider.LookupURLX(g.request.URL)
+ for _, strategy := range authStrategies {
+ strategy.ApplyOnRR(g.request)
}
}
if g.rawRequest != nil {
@@ -101,11 +142,11 @@ func (g *generatedRequest) ApplyAuth(provider authprovider.AuthProvider) {
gologger.Warning().Msgf("[authprovider] Could not parse URL %s: %s\n", g.rawRequest.FullURL, err)
return
}
- auth := provider.LookupURLX(parsed)
- if auth != nil {
- // here we need to apply it custom because we don't have a standard/official
- // rawhttp request format ( which we probably should have )
- g.rawRequest.ApplyAuthStrategy(auth)
+ authStrategies := provider.LookupURLX(parsed)
+ // here we need to apply it custom because we don't have a standard/official
+ // rawhttp request format ( which we probably should have )
+ for _, strategy := range authStrategies {
+ g.rawRequest.ApplyAuthStrategy(strategy)
}
}
}
@@ -204,7 +245,7 @@ func (r *requestGenerator) Make(ctx context.Context, input *contextargs.Context,
finalVars := generators.MergeMaps(allVars, payloads)
if vardump.EnableVarDump {
- gologger.Debug().Msgf("HTTP Protocol request variables: \n%s\n", vardump.DumpVariables(finalVars))
+ gologger.Debug().Msgf("HTTP Protocol request variables: %s\n", vardump.DumpVariables(finalVars))
}
// Note: If possible any changes to current logic (i.e evaluate -> then parse URL)
diff --git a/pkg/protocols/http/build_request_test.go b/pkg/protocols/http/build_request_test.go
index 401798a26d..1e965bf880 100644
--- a/pkg/protocols/http/build_request_test.go
+++ b/pkg/protocols/http/build_request_test.go
@@ -7,12 +7,30 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/interactsh"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/build_request_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/interactsh"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/build_request_test.go
+>>>>>>> projectdiscovery-main
)
func TestMakeRequestFromModal(t *testing.T) {
diff --git a/pkg/protocols/http/cluster.go b/pkg/protocols/http/cluster.go
index 3839817454..5fe2ad4035 100644
--- a/pkg/protocols/http/cluster.go
+++ b/pkg/protocols/http/cluster.go
@@ -1,30 +1,26 @@
package http
import (
+<<<<<<< HEAD
sliceutil "github.com/projectdiscovery/utils/slice"
"golang.org/x/exp/maps"
+=======
+ "fmt"
+ "strings"
+
+ "github.com/cespare/xxhash"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+>>>>>>> projectdiscovery-main
)
-// CanCluster returns true if the request can be clustered.
-//
-// This used by the clustering engine to decide whether two requests
-// are similar enough to be considered one and can be checked by
-// just adding the matcher/extractors for the request and the correct IDs.
-func (request *Request) CanCluster(other *Request) bool {
- if len(request.Payloads) > 0 || len(request.Fuzzing) > 0 || len(request.Raw) > 0 || len(request.Body) > 0 || request.Unsafe || request.NeedsRequestCondition() || request.Name != "" {
- return false
- }
- if request.Method != other.Method ||
- request.MaxRedirects != other.MaxRedirects ||
- request.DisableCookie != other.DisableCookie ||
- request.Redirects != other.Redirects {
- return false
- }
- if !sliceutil.Equal(request.Path, other.Path) {
- return false
- }
- if !maps.Equal(request.Headers, other.Headers) {
- return false
- }
- return true
+// TmplClusterKey generates a unique key for the request
+// to be used in the clustering process.
+func (request *Request) TmplClusterKey() uint64 {
+ inp := fmt.Sprintf("%s-%d-%t-%t-%s-%d", request.Method.String(), request.MaxRedirects, request.DisableCookie, request.Redirects, strings.Join(request.Path, "-"), utils.MapHash(request.Headers))
+ return xxhash.Sum64String(inp)
+}
+
+// IsClusterable returns true if the request is eligible to be clustered.
+func (request *Request) IsClusterable() bool {
+ return !(len(request.Payloads) > 0 || len(request.Fuzzing) > 0 || len(request.Raw) > 0 || len(request.Body) > 0 || request.Unsafe || request.NeedsRequestCondition() || request.Name != "")
}
diff --git a/pkg/protocols/http/cluster_test.go b/pkg/protocols/http/cluster_test.go
index a16db41ee0..da9c1e5a76 100644
--- a/pkg/protocols/http/cluster_test.go
+++ b/pkg/protocols/http/cluster_test.go
@@ -8,8 +8,11 @@ import (
func TestCanCluster(t *testing.T) {
req := &Request{Unsafe: true}
- require.False(t, req.CanCluster(&Request{}), "could cluster unsafe request")
+ require.False(t, req.IsClusterable(), "could cluster unsafe request")
req = &Request{Path: []string{"{{BaseURL}}"}, Method: HTTPMethodTypeHolder{MethodType: HTTPGet}}
- require.True(t, req.CanCluster(&Request{Path: []string{"{{BaseURL}}"}, Method: HTTPMethodTypeHolder{MethodType: HTTPGet}}), "could not cluster GET request")
+ newReq := &Request{Path: []string{"{{BaseURL}}"}, Method: HTTPMethodTypeHolder{MethodType: HTTPGet}}
+ require.True(t, req.IsClusterable(), "could not cluster GET request")
+ require.True(t, req.IsClusterable(), "could not cluster GET request")
+ require.Equal(t, req.TmplClusterKey(), newReq.TmplClusterKey(), "cluster keys should be equal")
}
diff --git a/pkg/protocols/http/http.go b/pkg/protocols/http/http.go
index 30db5000cb..f068c96e96 100644
--- a/pkg/protocols/http/http.go
+++ b/pkg/protocols/http/http.go
@@ -3,12 +3,15 @@ package http
import (
"bytes"
"fmt"
+ "math"
"strings"
+ "time"
"github.com/invopop/jsonschema"
json "github.com/json-iterator/go"
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/fuzz"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/operators/matchers"
@@ -19,6 +22,39 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/http/httpclientpool"
httputil "github.com/Explorer1092/nuclei/v3/pkg/protocols/utils/http"
"github.com/Explorer1092/nuclei/v3/pkg/utils/stats"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/http.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http/fuzz"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http/httpclientpool"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/fuzz"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/httpclientpool"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ _ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz/analyzers/time"
+
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz"
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz/analyzers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/httpclientpool"
+ httputil "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils/http"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/stats"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/http.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/rawhttp"
"github.com/projectdiscovery/retryablehttp-go"
fileutil "github.com/projectdiscovery/utils/file"
@@ -126,6 +162,9 @@ type Request struct {
// Fuzzing describes schema to fuzz http requests
Fuzzing []*fuzz.Rule `yaml:"fuzzing,omitempty" json:"fuzzing,omitempty" jsonschema:"title=fuzzin rules for http fuzzing,description=Fuzzing describes rule schema to fuzz http requests"`
+ // description: |
+ // Analyzer is an analyzer to use for matching the response.
+ Analyzer *analyzers.AnalyzerTemplate `yaml:"analyzer,omitempty" json:"analyzer,omitempty" jsonschema:"title=analyzer for http request,description=Analyzer for HTTP Request"`
CompiledOperators *operators.Operators `yaml:"-" json:"-"`
@@ -147,6 +186,10 @@ type Request struct {
// - "AWS"
Signature SignatureTypeHolder `yaml:"signature,omitempty" json:"signature,omitempty" jsonschema:"title=signature is the http request signature method,description=Signature is the HTTP Request signature Method,enum=AWS"`
+ // description: |
+ // SkipSecretFile skips the authentication or authorization configured in the secret file.
+ SkipSecretFile bool `yaml:"skip-secret-file,omitempty" json:"skip-secret-file,omitempty" jsonschema:"title=bypass secret file,description=Skips the authentication or authorization configured in the secret file"`
+
// description: |
// CookieReuse is an optional setting that enables cookie reuse for
// all requests defined in raw section.
@@ -219,6 +262,9 @@ type Request struct {
// FuzzPreConditionOperator is the operator between multiple PreConditions for fuzzing Default is OR
FuzzPreConditionOperator string `yaml:"pre-condition-operator,omitempty" json:"pre-condition-operator,omitempty" jsonschema:"title=condition between the filters,description=Operator to use between multiple per-conditions,enum=and,enum=or"`
fuzzPreConditionOperator matchers.ConditionType `yaml:"-" json:"-"`
+ // description: |
+ // GlobalMatchers marks matchers as static and applies globally to all result events from other templates
+ GlobalMatchers bool `yaml:"global-matchers,omitempty" json:"global-matchers,omitempty" jsonschema:"title=global matchers,description=marks matchers as static and applies globally to all result events from other templates"`
}
func (e Request) JSONSchemaExtend(schema *jsonschema.Schema) {
@@ -296,6 +342,21 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
},
RedirectFlow: httpclientpool.DontFollowRedirect,
}
+ var customTimeout int
+ if request.Analyzer != nil && request.Analyzer.Name == "time_delay" {
+ var timeoutVal int
+ if timeout, ok := request.Analyzer.Parameters["sleep_duration"]; ok {
+ timeoutVal, _ = timeout.(int)
+ } else {
+ timeoutVal = 5
+ }
+
+ // Add 3x buffer to the timeout
+ customTimeout = int(math.Ceil(float64(timeoutVal) * 3))
+ }
+ if customTimeout > 0 {
+ connectionConfiguration.Connection.CustomMaxTimeout = time.Duration(customTimeout) * time.Second
+ }
if request.Redirects || options.Options.FollowRedirects {
connectionConfiguration.RedirectFlow = httpclientpool.FollowAllRedirect
@@ -336,7 +397,7 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
request.Raw[i] = strings.ReplaceAll(raw, "\n", "\r\n")
}
}
- request.rawhttpClient = httpclientpool.GetRawHTTP(options.Options)
+ request.rawhttpClient = httpclientpool.GetRawHTTP(options)
}
if len(request.Matchers) > 0 || len(request.Extractors) > 0 {
compiled := &request.Operators
@@ -362,6 +423,12 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
}
}
+ if request.Analyzer != nil {
+ if analyzer := analyzers.GetAnalyzer(request.Analyzer.Name); analyzer == nil {
+ return errors.Errorf("analyzer %s not found", request.Analyzer.Name)
+ }
+ }
+
// Resolve payload paths from vars if they exists
for name, payload := range request.options.Options.Vars.AsMap() {
payloadStr, ok := payload.(string)
diff --git a/pkg/protocols/http/http_test.go b/pkg/protocols/http/http_test.go
index 3e406a46ca..3b0c0a19e5 100644
--- a/pkg/protocols/http/http_test.go
+++ b/pkg/protocols/http/http_test.go
@@ -5,10 +5,24 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/http_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/http_test.go
+>>>>>>> projectdiscovery-main
)
func TestHTTPCompile(t *testing.T) {
diff --git a/pkg/protocols/http/httpclientpool/clientpool.go b/pkg/protocols/http/httpclientpool/clientpool.go
index 4c357db58f..26868729c9 100644
--- a/pkg/protocols/http/httpclientpool/clientpool.go
+++ b/pkg/protocols/http/httpclientpool/clientpool.go
@@ -16,37 +16,44 @@ import (
"golang.org/x/net/proxy"
"golang.org/x/net/publicsuffix"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/utils"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/Explorer1092/nuclei/v3/pkg/types/scanstrategy"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/httpclientpool/clientpool.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/fastdialer/fastdialer"
"github.com/projectdiscovery/fastdialer/fastdialer/ja3/impersonate"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/fastdialer/fastdialer/ja3/impersonate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types/scanstrategy"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/httpclientpool/clientpool.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/rawhttp"
"github.com/projectdiscovery/retryablehttp-go"
mapsutil "github.com/projectdiscovery/utils/maps"
)
var (
- // Dialer is a copy of the fastdialer from protocolstate
- Dialer *fastdialer.Dialer
-
rawHttpClient *rawhttp.Client
+ rawHttpClientOnce sync.Once
forceMaxRedirects int
normalClient *retryablehttp.Client
clientPool *mapsutil.SyncLockMap[string, *retryablehttp.Client]
- // ResponseHeaderTimeout is the timeout for response headers
- // to be read from the server (this prevents infinite hang started by server if any)
- ResponseHeaderTimeout = time.Duration(5) * time.Second
- // HttpTimeoutMultiplier is the multiplier for the http timeout
- HttpTimeoutMultiplier = 3
)
-// GetHttpTimeout returns the http timeout for the client
-func GetHttpTimeout(opts *types.Options) time.Duration {
- return time.Duration(opts.Timeout*HttpTimeoutMultiplier) * time.Second
-}
-
// Init initializes the clientpool implementation
func Init(options *types.Options) error {
// Don't create clients if already created in the past.
@@ -72,6 +79,9 @@ func Init(options *types.Options) error {
type ConnectionConfiguration struct {
// DisableKeepAlive of the connection
DisableKeepAlive bool
+ // CustomMaxTimeout is the custom timeout for the connection
+ // This overrides all other timeouts and is used for accurate time based fuzzing.
+ CustomMaxTimeout time.Duration
cookiejar *cookiejar.Jar
mu sync.RWMutex
}
@@ -111,6 +121,24 @@ type Configuration struct {
RedirectFlow RedirectFlow
// Connection defines custom connection configuration
Connection *ConnectionConfiguration
+ // ResponseHeaderTimeout is the timeout for response body to be read from the server
+ ResponseHeaderTimeout time.Duration
+}
+
+func (c *Configuration) Clone() *Configuration {
+ clone := *c
+ if c.Connection != nil {
+ cloneConnection := &ConnectionConfiguration{
+ DisableKeepAlive: c.Connection.DisableKeepAlive,
+ }
+ if c.Connection.HasCookieJar() {
+ cookiejar := *c.Connection.GetCookieJar()
+ cloneConnection.SetCookieJar(&cookiejar)
+ }
+ clone.Connection = cloneConnection
+ }
+
+ return &clone
}
// Hash returns the hash of the configuration to allow client pooling
@@ -129,29 +157,35 @@ func (c *Configuration) Hash() string {
builder.WriteString(strconv.FormatBool(c.DisableCookie))
builder.WriteString("c")
builder.WriteString(strconv.FormatBool(c.Connection != nil))
+ if c.Connection != nil && c.Connection.CustomMaxTimeout > 0 {
+ builder.WriteString("k")
+ builder.WriteString(c.Connection.CustomMaxTimeout.String())
+ }
+ builder.WriteString("r")
+ builder.WriteString(strconv.FormatInt(int64(c.ResponseHeaderTimeout.Seconds()), 10))
hash := builder.String()
return hash
}
// HasStandardOptions checks whether the configuration requires custom settings
func (c *Configuration) HasStandardOptions() bool {
- return c.Threads == 0 && c.MaxRedirects == 0 && c.RedirectFlow == DontFollowRedirect && c.DisableCookie && c.Connection == nil && !c.NoTimeout
+ return c.Threads == 0 && c.MaxRedirects == 0 && c.RedirectFlow == DontFollowRedirect && c.DisableCookie && c.Connection == nil && !c.NoTimeout && c.ResponseHeaderTimeout == 0
}
// GetRawHTTP returns the rawhttp request client
-func GetRawHTTP(options *types.Options) *rawhttp.Client {
- if rawHttpClient == nil {
+func GetRawHTTP(options *protocols.ExecutorOptions) *rawhttp.Client {
+ rawHttpClientOnce.Do(func() {
rawHttpOptions := rawhttp.DefaultOptions
if types.ProxyURL != "" {
rawHttpOptions.Proxy = types.ProxyURL
} else if types.ProxySocksURL != "" {
rawHttpOptions.Proxy = types.ProxySocksURL
- } else if Dialer != nil {
- rawHttpOptions.FastDialer = Dialer
+ } else if protocolstate.Dialer != nil {
+ rawHttpOptions.FastDialer = protocolstate.Dialer
}
- rawHttpOptions.Timeout = GetHttpTimeout(options)
+ rawHttpOptions.Timeout = options.Options.GetTimeouts().HttpTimeout
rawHttpClient = rawhttp.NewClient(rawHttpOptions)
- }
+ })
return rawHttpClient
}
@@ -167,10 +201,6 @@ func Get(options *types.Options, configuration *Configuration) (*retryablehttp.C
func wrappedGet(options *types.Options, configuration *Configuration) (*retryablehttp.Client, error) {
var err error
- if Dialer == nil {
- Dialer = protocolstate.Dialer
- }
-
hash := configuration.Hash()
if client, ok := clientPool.Get(hash); ok {
return client, nil
@@ -182,6 +212,9 @@ func wrappedGet(options *types.Options, configuration *Configuration) (*retryabl
maxIdleConns := 0
maxConnsPerHost := 0
maxIdleConnsPerHost := -1
+ // do not split given timeout into chunks for retry
+ // because this won't work on slow hosts
+ retryableHttpOptions.NoAdjustTimeout = true
if configuration.Threads > 0 || options.ScanStrategy == scanstrategy.HostSpray.String() {
// Single host
@@ -235,24 +268,33 @@ func wrappedGet(options *types.Options, configuration *Configuration) (*retryabl
return nil, errors.Wrap(err, "could not create client certificate")
}
+ // responseHeaderTimeout is max timeout for response headers to be read
+ responseHeaderTimeout := options.GetTimeouts().HttpResponseHeaderTimeout
+ if configuration.ResponseHeaderTimeout != 0 {
+ responseHeaderTimeout = configuration.ResponseHeaderTimeout
+ }
+ if configuration.Connection != nil && configuration.Connection.CustomMaxTimeout > 0 {
+ responseHeaderTimeout = configuration.Connection.CustomMaxTimeout
+ }
+
transport := &http.Transport{
ForceAttemptHTTP2: options.ForceAttemptHTTP2,
- DialContext: Dialer.Dial,
+ DialContext: protocolstate.GetDialer().Dial,
DialTLSContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
if options.TlsImpersonate {
- return Dialer.DialTLSWithConfigImpersonate(ctx, network, addr, tlsConfig, impersonate.Random, nil)
+ return protocolstate.Dialer.DialTLSWithConfigImpersonate(ctx, network, addr, tlsConfig, impersonate.Random, nil)
}
if options.HasClientCertificates() || options.ForceAttemptHTTP2 {
- return Dialer.DialTLSWithConfig(ctx, network, addr, tlsConfig)
+ return protocolstate.Dialer.DialTLSWithConfig(ctx, network, addr, tlsConfig)
}
- return Dialer.DialTLS(ctx, network, addr)
+ return protocolstate.GetDialer().DialTLS(ctx, network, addr)
},
MaxIdleConns: maxIdleConns,
MaxIdleConnsPerHost: maxIdleConnsPerHost,
MaxConnsPerHost: maxConnsPerHost,
TLSClientConfig: tlsConfig,
DisableKeepAlives: disableKeepAlives,
- ResponseHeaderTimeout: ResponseHeaderTimeout,
+ ResponseHeaderTimeout: responseHeaderTimeout,
}
if types.ProxyURL != "" {
@@ -299,7 +341,10 @@ func wrappedGet(options *types.Options, configuration *Configuration) (*retryabl
CheckRedirect: makeCheckRedirectFunc(redirectFlow, maxRedirects),
}
if !configuration.NoTimeout {
- httpclient.Timeout = GetHttpTimeout(options)
+ httpclient.Timeout = options.GetTimeouts().HttpTimeout
+ if configuration.Connection != nil && configuration.Connection.CustomMaxTimeout > 0 {
+ httpclient.Timeout = configuration.Connection.CustomMaxTimeout
+ }
}
client := retryablehttp.NewWithHTTPClient(httpclient, retryableHttpOptions)
if jar != nil {
@@ -351,7 +396,7 @@ func makeCheckRedirectFunc(redirectType RedirectFlow, maxRedirects int) checkRed
}
}
-func checkMaxRedirects(req *http.Request, via []*http.Request, maxRedirects int) error {
+func checkMaxRedirects(_ *http.Request, via []*http.Request, maxRedirects int) error {
if maxRedirects == 0 {
if len(via) > defaultMaxRedirects {
return http.ErrUseLastResponse
diff --git a/pkg/protocols/http/httpclientpool/options.go b/pkg/protocols/http/httpclientpool/options.go
new file mode 100644
index 0000000000..92e42fe96b
--- /dev/null
+++ b/pkg/protocols/http/httpclientpool/options.go
@@ -0,0 +1,76 @@
+package httpclientpool
+
+import (
+ "io"
+ "net/http"
+ "strings"
+ "time"
+
+ "github.com/projectdiscovery/rawhttp"
+ stringsutil "github.com/projectdiscovery/utils/strings"
+ urlutil "github.com/projectdiscovery/utils/url"
+)
+
+// WithCustomTimeout is a configuration for custom timeout
+type WithCustomTimeout struct {
+ Timeout time.Duration
+}
+
+// RawHttpRequestOpts is a configuration for raw http request
+type RawHttpRequestOpts struct {
+ // Method is the http method to use
+ Method string
+ // URL is the url to request
+ URL string
+ // Path is request path to use
+ Path string
+ // Headers is the headers to use
+ Headers map[string][]string
+ // Body is the body to use
+ Body io.Reader
+ // Options is more client related options
+ Options *rawhttp.Options
+}
+
+// SendRawRequest sends a raw http request with the provided options and returns http response
+func SendRawRequest(client *rawhttp.Client, opts *RawHttpRequestOpts) (*http.Response, error) {
+ resp, err := client.DoRawWithOptions(opts.Method, opts.URL, opts.Path, opts.Headers, opts.Body, opts.Options)
+ if err != nil {
+ cause := err.Error()
+ if stringsutil.ContainsAll(cause, "ReadStatusLine: ", "read: connection reset by peer") {
+ // this error is caused when rawhttp client sends a corrupted or malformed request packet to server
+ // some servers may attempt gracefully shutdown but most will just abruptly close the connection which results
+ // in a connection reset by peer error and this can be safely assumed as 400 Bad Request in terms of normal http flow
+ req, reqErr := http.NewRequest(opts.Method, opts.URL, opts.Body)
+ if reqErr != nil {
+ // failed to build new request mostly because of invalid url or body
+ // try again or else return urlErr
+ parsed, urlErr := urlutil.ParseAbsoluteURL(opts.URL, true)
+ if urlErr != nil {
+ return nil, err
+ }
+ req, reqErr = http.NewRequest(opts.Method, parsed.Host, opts.Body)
+ if reqErr != nil {
+ return nil, err
+ }
+ req.URL = parsed.URL
+ req.Header = opts.Headers
+ }
+
+ // if req is still nil, return error
+ if req == nil {
+ return nil, err
+ }
+
+ req.Header = opts.Headers
+ resp = &http.Response{
+ Request: req,
+ StatusCode: http.StatusBadRequest,
+ Status: http.StatusText(http.StatusBadRequest),
+ Body: io.NopCloser(strings.NewReader("")),
+ }
+ return resp, nil
+ }
+ }
+ return resp, err
+}
diff --git a/pkg/protocols/http/httputils/spm.go b/pkg/protocols/http/httputils/spm.go
index bb0b337c77..22fdd8f93b 100644
--- a/pkg/protocols/http/httputils/spm.go
+++ b/pkg/protocols/http/httputils/spm.go
@@ -123,9 +123,11 @@ func (h *StopAtFirstMatchHandler[T]) MatchCallback(fn func()) {
// run runs the internal handler
func (h *StopAtFirstMatchHandler[T]) run(ctx context.Context) {
defer h.internalWg.Done()
+
for {
select {
case <-ctx.Done():
+ return
case val, ok := <-h.ResultChan:
if !ok {
return
diff --git a/pkg/protocols/http/operators.go b/pkg/protocols/http/operators.go
index 5742a9745b..173772f3f5 100644
--- a/pkg/protocols/http/operators.go
+++ b/pkg/protocols/http/operators.go
@@ -5,6 +5,7 @@ import (
"strings"
"time"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/operators/extractors"
@@ -14,6 +15,29 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/utils"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/operators.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/utils"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/operators.go
+>>>>>>> projectdiscovery-main
)
// Match matches a generic data response again a given matcher
@@ -166,10 +190,19 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
if types.ToString(wrapped.InternalEvent["path"]) != "" {
fields.Path = types.ToString(wrapped.InternalEvent["path"])
}
+ var isGlobalMatchers bool
+ if value, ok := wrapped.InternalEvent["global-matchers"]; ok {
+ isGlobalMatchers = value.(bool)
+ }
+ var analyzerDetails string
+ if value, ok := wrapped.InternalEvent["analyzer_details"]; ok {
+ analyzerDetails = value.(string)
+ }
data := &output.ResultEvent{
TemplateID: types.ToString(wrapped.InternalEvent["template-id"]),
TemplatePath: types.ToString(wrapped.InternalEvent["template-path"]),
Info: wrapped.InternalEvent["template-info"].(model.Info),
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Host: fields.Host,
Port: fields.Port,
@@ -182,11 +215,13 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
Timestamp: time.Now(),
MatcherStatus: true,
IP: fields.Ip,
+ GlobalMatchers: isGlobalMatchers,
Request: types.ToString(wrapped.InternalEvent["request"]),
Response: request.truncateResponse(wrapped.InternalEvent["response"]),
CURLCommand: types.ToString(wrapped.InternalEvent["curl-command"]),
TemplateEncoded: request.options.EncodeTemplate(),
Error: types.ToString(wrapped.InternalEvent["error"]),
+ AnalyzerDetails: analyzerDetails,
}
return data
}
diff --git a/pkg/protocols/http/operators_test.go b/pkg/protocols/http/operators_test.go
index bfcf686ab5..40f3bbbaeb 100644
--- a/pkg/protocols/http/operators_test.go
+++ b/pkg/protocols/http/operators_test.go
@@ -7,6 +7,7 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
@@ -14,6 +15,25 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/operators/matchers"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/operators_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/operators_test.go
+>>>>>>> projectdiscovery-main
)
func TestResponseToDSLMap(t *testing.T) {
diff --git a/pkg/protocols/http/request.go b/pkg/protocols/http/request.go
index 770dbfdba9..be05ca0de3 100644
--- a/pkg/protocols/http/request.go
+++ b/pkg/protocols/http/request.go
@@ -35,13 +35,36 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/fastdialer/fastdialer"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz/analyzers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/httpclientpool"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/httputils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/signer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/signerpool"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types/nucleierr"
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/rawhttp"
convUtil "github.com/projectdiscovery/utils/conversion"
+ "github.com/projectdiscovery/utils/errkit"
errorutil "github.com/projectdiscovery/utils/errors"
httpUtils "github.com/projectdiscovery/utils/http"
"github.com/projectdiscovery/utils/reader"
sliceutil "github.com/projectdiscovery/utils/slice"
stringsutil "github.com/projectdiscovery/utils/strings"
+ unitutils "github.com/projectdiscovery/utils/unit"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -53,9 +76,11 @@ const (
)
var (
- MaxBodyRead = int64(10 * 1024 * 1024) // 10MB
+ MaxBodyRead = 10 * unitutils.Mega
// ErrMissingVars is error occured when variables are missing
- ErrMissingVars = errors.New("stop execution due to unresolved variables")
+ ErrMissingVars = errkit.New("stop execution due to unresolved variables").SetKind(nucleierr.ErrTemplateLogic).Build()
+ // ErrHttpEngineRequestDeadline is error occured when request deadline set by http request engine is exceeded
+ ErrHttpEngineRequestDeadline = errkit.New("http request engine deadline exceeded").SetKind(errkit.ErrKindDeadline).Build()
)
// Type returns the type of the protocol request
@@ -114,7 +139,13 @@ func (request *Request) executeRaceRequest(input *contextargs.Context, previous
}
shouldStop := (request.options.Options.StopAtFirstMatch || request.StopAtFirstMatch || request.options.StopAtFirstMatch)
- spmHandler := httputils.NewNonBlockingSPMHandler[error](ctx, maxErrorsWhenParallel, shouldStop)
+
+ childCtx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+
+ spmHandler := httputils.NewNonBlockingSPMHandler[error](childCtx, maxErrorsWhenParallel, shouldStop)
+ defer spmHandler.Cancel()
+
gotMatches := &atomic.Bool{}
// wrappedCallback is a callback that wraps the original callback
// to implement stop at first match logic
@@ -141,15 +172,15 @@ func (request *Request) executeRaceRequest(input *contextargs.Context, previous
return
}
// marks thsi host as unresponsive if applicable
- request.markUnresponsiveHost(input, err)
- if request.isUnresponsiveHost(input) {
+ request.markUnresponsiveAddress(input, err)
+ if request.isUnresponsiveAddress(input) {
// stop all inflight requests
spmHandler.Cancel()
}
})
for i := 0; i < request.RaceNumberRequests; i++ {
- if spmHandler.FoundFirstMatch() || request.isUnresponsiveHost(input) {
+ if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(input) {
// stop sending more requests condition is met
break
}
@@ -157,7 +188,7 @@ func (request *Request) executeRaceRequest(input *contextargs.Context, previous
// execute http request
go func(httpRequest *generatedRequest) {
defer spmHandler.Release()
- if spmHandler.FoundFirstMatch() || request.isUnresponsiveHost(input) {
+ if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(input) {
// stop sending more requests condition is met
return
}
@@ -195,7 +226,13 @@ func (request *Request) executeParallelHTTP(input *contextargs.Context, dynamicV
// Stop-at-first-match logic while executing requests
// parallely using threads
shouldStop := (request.options.Options.StopAtFirstMatch || request.StopAtFirstMatch || request.options.StopAtFirstMatch)
- spmHandler := httputils.NewBlockingSPMHandler[error](context.Background(), maxWorkers, maxErrorsWhenParallel, shouldStop)
+
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+
+ spmHandler := httputils.NewBlockingSPMHandler[error](ctx, maxWorkers, maxErrorsWhenParallel, shouldStop)
+ defer spmHandler.Cancel()
+
// wrappedCallback is a callback that wraps the original callback
// to implement stop at first match logic
wrappedCallback := func(event *output.InternalWrappedEvent) {
@@ -220,8 +257,8 @@ func (request *Request) executeParallelHTTP(input *contextargs.Context, dynamicV
return
}
// marks thsi host as unresponsive if applicable
- request.markUnresponsiveHost(input, err)
- if request.isUnresponsiveHost(input) {
+ request.markUnresponsiveAddress(input, err)
+ if request.isUnresponsiveAddress(input) {
// stop all inflight requests
spmHandler.Cancel()
}
@@ -249,7 +286,7 @@ func (request *Request) executeParallelHTTP(input *contextargs.Context, dynamicV
}
// break if stop at first match is found or host is unresponsive
- if spmHandler.FoundFirstMatch() || request.isUnresponsiveHost(input) {
+ if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(input) {
break
}
@@ -265,17 +302,23 @@ func (request *Request) executeParallelHTTP(input *contextargs.Context, dynamicV
if input.MetaInput.Input == "" {
input.MetaInput.Input = generatedHttpRequest.URL()
}
+ updatedInput := contextargs.GetCopyIfHostOutdated(input, generatedHttpRequest.URL())
+ if request.isUnresponsiveAddress(updatedInput) {
+ // skip on unresponsive host no need to continue
+ spmHandler.Cancel()
+ return nil
+ }
spmHandler.Acquire()
go func(httpRequest *generatedRequest) {
defer spmHandler.Release()
- if spmHandler.FoundFirstMatch() || request.isUnresponsiveHost(input) || spmHandler.Cancelled() {
+ if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(updatedInput) || spmHandler.Cancelled() {
return
}
// putting ratelimiter here prevents any unnecessary waiting if any
request.options.RateLimitTake()
// after ratelimit take, check if we need to stop
- if spmHandler.FoundFirstMatch() || request.isUnresponsiveHost(input) || spmHandler.Cancelled() {
+ if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(updatedInput) || spmHandler.Cancelled() {
return
}
@@ -327,7 +370,13 @@ func (request *Request) executeTurboHTTP(input *contextargs.Context, dynamicValu
// Stop-at-first-match logic while executing requests
// parallely using threads
shouldStop := (request.options.Options.StopAtFirstMatch || request.StopAtFirstMatch || request.options.StopAtFirstMatch)
- spmHandler := httputils.NewBlockingSPMHandler[error](context.Background(), maxWorkers, maxErrorsWhenParallel, shouldStop)
+
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+
+ spmHandler := httputils.NewBlockingSPMHandler[error](ctx, maxWorkers, maxErrorsWhenParallel, shouldStop)
+ defer spmHandler.Cancel()
+
// wrappedCallback is a callback that wraps the original callback
// to implement stop at first match logic
wrappedCallback := func(event *output.InternalWrappedEvent) {
@@ -352,8 +401,8 @@ func (request *Request) executeTurboHTTP(input *contextargs.Context, dynamicValu
return
}
// marks thsi host as unresponsive if applicable
- request.markUnresponsiveHost(input, err)
- if request.isUnresponsiveHost(input) {
+ request.markUnresponsiveAddress(input, err)
+ if request.isUnresponsiveAddress(input) {
// stop all inflight requests
spmHandler.Cancel()
}
@@ -371,7 +420,7 @@ func (request *Request) executeTurboHTTP(input *contextargs.Context, dynamicValu
default:
}
- if spmHandler.FoundFirstMatch() || request.isUnresponsiveHost(input) || spmHandler.Cancelled() {
+ if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(input) || spmHandler.Cancelled() {
// skip if first match is found
break
}
@@ -385,11 +434,17 @@ func (request *Request) executeTurboHTTP(input *contextargs.Context, dynamicValu
if input.MetaInput.Input == "" {
input.MetaInput.Input = generatedHttpRequest.URL()
}
+ updatedInput := contextargs.GetCopyIfHostOutdated(input, generatedHttpRequest.URL())
+ if request.isUnresponsiveAddress(updatedInput) {
+ // skip on unresponsive host no need to continue
+ spmHandler.Cancel()
+ return nil
+ }
generatedHttpRequest.pipelinedClient = pipeClient
spmHandler.Acquire()
go func(httpRequest *generatedRequest) {
defer spmHandler.Release()
- if spmHandler.FoundFirstMatch() || request.isUnresponsiveHost(input) {
+ if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(updatedInput) {
// skip if first match is found
return
}
@@ -448,7 +503,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
request.options.RateLimitTake()
ctx := request.newContext(input)
- ctxWithTimeout, cancel := context.WithTimeout(ctx, httpclientpool.GetHttpTimeout(request.options.Options))
+ ctxWithTimeout, cancel := context.WithTimeoutCause(ctx, request.options.Options.GetTimeouts().HttpTimeout, ErrHttpEngineRequestDeadline)
defer cancel()
generatedHttpRequest, err := generator.Make(ctxWithTimeout, input, data, payloads, dynamicValue)
@@ -459,6 +514,12 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
request.options.Progress.IncrementFailedRequestsBy(int64(generator.Total()))
return true, err
}
+ // ideally if http template used a custom port or hostname
+ // we would want to update it in input but currently templateCtx logic
+ // is closely tied to contextargs.Context so we are temporarily creating
+ // a copy and using it to check for host errors etc
+ // but this should be replaced once templateCtx is refactored properly
+ updatedInput := contextargs.GetCopyIfHostOutdated(input, generatedHttpRequest.URL())
if generatedHttpRequest.customCancelFunction != nil {
defer generatedHttpRequest.customCancelFunction()
@@ -469,7 +530,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
input.MetaInput.Input = generatedHttpRequest.URL()
}
// Check if hosts keep erroring
- if request.isUnresponsiveHost(input) {
+ if request.isUnresponsiveAddress(updatedInput) {
return true, nil
}
var gotMatches bool
@@ -509,7 +570,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
}
if execReqErr != nil {
// if applicable mark the host as unresponsive
- request.markUnresponsiveHost(input, execReqErr)
+ request.markUnresponsiveAddress(updatedInput, execReqErr)
requestErr = errorutil.NewWithErr(execReqErr).Msgf("got err while executing %v", generatedHttpRequest.URL())
request.options.Progress.IncrementFailedRequestsBy(1)
} else {
@@ -557,12 +618,12 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
return requestErr
}
-const drainReqSize = int64(8 * 1024)
+const drainReqSize = int64(8 * unitutils.Kilo)
// executeRequest executes the actual generated request and returns error if occurred
func (request *Request) executeRequest(input *contextargs.Context, generatedRequest *generatedRequest, previousEvent output.InternalEvent, hasInteractMatchers bool, processEvent protocols.OutputEventCallback, requestCount int) (err error) {
// Check if hosts keep erroring
- if request.isUnresponsiveHost(input) {
+ if request.isUnresponsiveAddress(input) {
return fmt.Errorf("hostErrorsCache : host %s is unresponsive", input.MetaInput.Input)
}
@@ -648,7 +709,7 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
}
// === apply auth strategies ===
- if generatedRequest.request != nil {
+ if generatedRequest.request != nil && !request.SkipSecretFile {
generatedRequest.ApplyAuth(request.options.AuthProvider)
}
@@ -695,7 +756,16 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
inputUrl = url.String()
}
formedURL = fmt.Sprintf("%s%s", inputUrl, generatedRequest.rawRequest.Path)
- resp, err = generatedRequest.original.rawhttpClient.DoRawWithOptions(generatedRequest.rawRequest.Method, inputUrl, generatedRequest.rawRequest.Path, generators.ExpandMapValues(generatedRequest.rawRequest.Headers), io.NopCloser(strings.NewReader(generatedRequest.rawRequest.Data)), &options)
+
+ // send rawhttp request and get response
+ resp, err = httpclientpool.SendRawRequest(generatedRequest.original.rawhttpClient, &httpclientpool.RawHttpRequestOpts{
+ Method: generatedRequest.rawRequest.Method,
+ URL: inputUrl,
+ Path: generatedRequest.rawRequest.Path,
+ Headers: generators.ExpandMapValues(generatedRequest.rawRequest.Headers),
+ Body: io.NopCloser(strings.NewReader(generatedRequest.rawRequest.Data)),
+ Options: &options,
+ })
} else {
//** For Normal requests **//
hostname = generatedRequest.request.URL.Host
@@ -713,17 +783,35 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
if errSignature := request.handleSignature(generatedRequest); errSignature != nil {
return errSignature
}
-
httpclient := request.httpClient
+
+ // this will be assigned/updated if this specific request has a custom configuration
+ var modifiedConfig *httpclientpool.Configuration
+
+ // check for cookie related configuration
if input.CookieJar != nil {
- connConfiguration := request.connConfiguration
+ connConfiguration := request.connConfiguration.Clone()
connConfiguration.Connection.SetCookieJar(input.CookieJar)
- client, err := httpclientpool.Get(request.options.Options, connConfiguration)
+ modifiedConfig = connConfiguration
+ }
+ // check for request updatedTimeout annotation
+ updatedTimeout, ok := generatedRequest.request.Context().Value(httpclientpool.WithCustomTimeout{}).(httpclientpool.WithCustomTimeout)
+ if ok {
+ if modifiedConfig == nil {
+ connConfiguration := request.connConfiguration.Clone()
+ modifiedConfig = connConfiguration
+ }
+ modifiedConfig.ResponseHeaderTimeout = updatedTimeout.Timeout
+ }
+
+ if modifiedConfig != nil {
+ client, err := httpclientpool.Get(request.options.Options, modifiedConfig)
if err != nil {
return errors.Wrap(err, "could not get http client")
}
httpclient = client
}
+
resp, err = httpclient.Do(generatedRequest.request)
}
}
@@ -776,7 +864,9 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
if input.MetaInput.CustomIP != "" {
outputEvent["ip"] = input.MetaInput.CustomIP
} else {
- outputEvent["ip"] = httpclientpool.Dialer.GetDialedIP(hostname)
+ outputEvent["ip"] = protocolstate.Dialer.GetDialedIP(hostname)
+ // try getting cname
+ request.addCNameIfAvailable(hostname, outputEvent)
}
if len(generatedRequest.interactshURLs) > 0 {
@@ -809,15 +899,15 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
// define max body read limit
maxBodylimit := MaxBodyRead // 10MB
if request.MaxSize > 0 {
- maxBodylimit = int64(request.MaxSize)
+ maxBodylimit = request.MaxSize
}
if request.options.Options.ResponseReadSize != 0 {
- maxBodylimit = int64(request.options.Options.ResponseReadSize)
+ maxBodylimit = request.options.Options.ResponseReadSize
}
// respChain is http response chain that reads response body
// efficiently by reusing buffers and does all decoding and optimizations
- respChain := httpUtils.NewResponseChain(resp, maxBodylimit)
+ respChain := httpUtils.NewResponseChain(resp, int64(maxBodylimit))
defer respChain.Close() // reuse buffers
// we only intend to log/save the final redirected response
@@ -857,8 +947,26 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
matchedURL = responseURL
}
}
+
finalEvent := make(output.InternalEvent)
+ if request.Analyzer != nil {
+ analyzer := analyzers.GetAnalyzer(request.Analyzer.Name)
+ analysisMatched, analysisDetails, err := analyzer.Analyze(&analyzers.Options{
+ FuzzGenerated: generatedRequest.fuzzGeneratedRequest,
+ HttpClient: request.httpClient,
+ ResponseTimeDelay: duration,
+ AnalyzerParameters: request.Analyzer.Parameters,
+ })
+ if err != nil {
+ gologger.Warning().Msgf("Could not analyze response: %v\n", err)
+ }
+ if analysisMatched {
+ finalEvent["analyzer_details"] = analysisDetails
+ finalEvent["analyzer"] = true
+ }
+ }
+
outputEvent := request.responseToDSLMap(respChain.Response(), input.MetaInput.Input, matchedURL, convUtil.String(dumpedRequest), respChain.FullResponse().String(), respChain.Body().String(), respChain.Headers().String(), duration, generatedRequest.meta)
// add response fields to template context and merge templatectx variables to output event
request.options.AddTemplateVars(input.MetaInput, request.Type(), request.ID, outputEvent)
@@ -872,7 +980,13 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
if input.MetaInput.CustomIP != "" {
outputEvent["ip"] = input.MetaInput.CustomIP
} else {
- outputEvent["ip"] = httpclientpool.Dialer.GetDialedIP(hostname)
+ dialer := protocolstate.GetDialer()
+ if dialer != nil {
+ outputEvent["ip"] = dialer.GetDialedIP(hostname)
+ }
+
+ // try getting cname
+ request.addCNameIfAvailable(hostname, outputEvent)
}
if request.options.Interactsh != nil {
request.options.Interactsh.MakePlaceholders(generatedRequest.interactshURLs, outputEvent)
@@ -897,13 +1011,22 @@ func (request *Request) executeRequest(input *contextargs.Context, generatedRequ
// prune signature internal values if any
request.pruneSignatureInternalValues(generatedRequest.meta)
- event := eventcreator.CreateEventWithAdditionalOptions(request, generators.MergeMaps(generatedRequest.dynamicValues, finalEvent), request.options.Options.Debug || request.options.Options.DebugResponse, func(internalWrappedEvent *output.InternalWrappedEvent) {
+ interimEvent := generators.MergeMaps(generatedRequest.dynamicValues, finalEvent)
+ isDebug := request.options.Options.Debug || request.options.Options.DebugResponse
+ event := eventcreator.CreateEventWithAdditionalOptions(request, interimEvent, isDebug, func(internalWrappedEvent *output.InternalWrappedEvent) {
internalWrappedEvent.OperatorsResult.PayloadValues = generatedRequest.meta
})
+
if hasInteractMatchers {
event.UsesInteractsh = true
}
+ if request.options.GlobalMatchers.HasMatchers() {
+ request.options.GlobalMatchers.Match(interimEvent, request.Match, request.Extract, isDebug, func(event output.InternalEvent, result *operators.Result) {
+ callback(eventcreator.CreateEventWithOperatorResults(request, event, result))
+ })
+ }
+
// if requrlpattern is enabled, only then it is reflected in result event else it is empty string
// consult @Ice3man543 before changing this logic (context: vuln_hash)
if request.options.ExportReqURLPattern {
@@ -964,6 +1087,27 @@ func (request *Request) validateNFixEvent(input *contextargs.Context, gr *genera
}
}
+// addCNameIfAvailable adds the cname to the event if available
+func (request *Request) addCNameIfAvailable(hostname string, outputEvent map[string]interface{}) {
+ if protocolstate.Dialer == nil {
+ return
+ }
+
+ data, err := protocolstate.Dialer.GetDNSData(hostname)
+ if err == nil {
+ switch len(data.CNAME) {
+ case 0:
+ return
+ case 1:
+ outputEvent["cname"] = data.CNAME[0]
+ default:
+ // add 1st and put others in cname_all
+ outputEvent["cname"] = data.CNAME[0]
+ outputEvent["cname_all"] = data.CNAME
+ }
+ }
+}
+
// handleSignature of the http request
func (request *Request) handleSignature(generatedRequest *generatedRequest) error {
switch request.Signature.Value {
@@ -995,10 +1139,15 @@ func (request *Request) setCustomHeaders(req *generatedRequest) {
req.rawRequest.Headers[k] = v
} else {
kk, vv := strings.TrimSpace(k), strings.TrimSpace(v)
- req.request.Header.Set(kk, vv)
+ // NOTE(dwisiswant0): Do we really not need to convert it first into
+ // lowercase?
if kk == "Host" {
req.request.Host = vv
+
+ continue
}
+
+ req.request.Header[kk] = []string{vv}
}
}
}
@@ -1069,20 +1218,20 @@ func (request *Request) newContext(input *contextargs.Context) context.Context {
return input.Context()
}
-// markUnresponsiveHost checks if the error is a unreponsive host error and marks it
-func (request *Request) markUnresponsiveHost(input *contextargs.Context, err error) {
+// markUnresponsiveAddress checks if the error is a unreponsive host error and marks it
+func (request *Request) markUnresponsiveAddress(input *contextargs.Context, err error) {
if err == nil {
return
}
if request.options.HostErrorsCache != nil {
- request.options.HostErrorsCache.MarkFailed(input.MetaInput.ID(), err)
+ request.options.HostErrorsCache.MarkFailed(request.options.ProtocolType.String(), input, err)
}
}
-// isUnresponsiveHost checks if the error is a unreponsive based on its execution history
-func (request *Request) isUnresponsiveHost(input *contextargs.Context) bool {
+// isUnresponsiveAddress checks if the error is a unreponsive based on its execution history
+func (request *Request) isUnresponsiveAddress(input *contextargs.Context) bool {
if request.options.HostErrorsCache != nil {
- return request.options.HostErrorsCache.Check(input.MetaInput.ID())
+ return request.options.HostErrorsCache.Check(request.options.ProtocolType.String(), input)
}
return false
}
diff --git a/pkg/protocols/http/request_annotations.go b/pkg/protocols/http/request_annotations.go
index ac47c6ab0b..724f833914 100644
--- a/pkg/protocols/http/request_annotations.go
+++ b/pkg/protocols/http/request_annotations.go
@@ -10,7 +10,13 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/http/httpclientpool"
"github.com/projectdiscovery/fastdialer/fastdialer"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/httpclientpool"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types/nucleierr"
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
+ "github.com/projectdiscovery/utils/errkit"
iputil "github.com/projectdiscovery/utils/ip"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -27,6 +33,15 @@ var (
reTimeoutAnnotation = regexp.MustCompile(`(?m)^@timeout:\s*(.+)\s*$`)
// @once sets the request to be executed only once for a specific URL
reOnceAnnotation = regexp.MustCompile(`(?m)^@once\s*$`)
+
+ // ErrTimeoutAnnotationDeadline is the error returned when a specific amount of time was exceeded for a request
+ // which was alloted using @timeout annotation this usually means that vulnerability was not found
+ // in rare case it could also happen due to network congestion
+ // the assigned class is TemplateLogic since this in almost every case means that server is not vulnerable
+ ErrTimeoutAnnotationDeadline = errkit.New("timeout annotation deadline exceeded").SetKind(nucleierr.ErrTemplateLogic).Build()
+ // ErrRequestTimeoutDeadline is the error returned when a specific amount of time was exceeded for a request
+ // this happens when the request execution exceeds alloted time
+ ErrRequestTimeoutDeadline = errkit.New("request timeout deadline exceeded when notimeout is set").SetKind(errkit.ErrKindDeadline).Build()
)
type flowMark int
@@ -119,13 +134,21 @@ func (r *Request) parseAnnotations(rawRequest string, request *retryablehttp.Req
if duration := reTimeoutAnnotation.FindStringSubmatch(rawRequest); len(duration) > 0 {
value := strings.TrimSpace(duration[1])
if parsed, err := time.ParseDuration(value); err == nil {
+ // to avoid dos via timeout request annotation in http template we set it to maximum of 2 minutes
+ if parsed > 2*time.Minute {
+ parsed = 2 * time.Minute
+ }
//nolint:govet // cancelled automatically by withTimeout
- ctx, overrides.cancelFunc = context.WithTimeout(context.Background(), parsed)
+ // global timeout is overridden by annotation by replacing context
+ ctx, overrides.cancelFunc = context.WithTimeoutCause(context.TODO(), parsed, ErrTimeoutAnnotationDeadline)
+ // add timeout value to context
+ ctx = context.WithValue(ctx, httpclientpool.WithCustomTimeout{}, httpclientpool.WithCustomTimeout{Timeout: parsed})
request = request.Clone(ctx)
}
} else {
//nolint:govet // cancelled automatically by withTimeout
- ctx, overrides.cancelFunc = context.WithTimeout(context.Background(), httpclientpool.GetHttpTimeout(r.options.Options))
+ // global timeout is overridden by annotation by replacing context
+ ctx, overrides.cancelFunc = context.WithTimeoutCause(context.TODO(), r.options.Options.GetTimeouts().HttpTimeout, ErrRequestTimeoutDeadline)
request = request.Clone(ctx)
}
}
diff --git a/pkg/protocols/http/request_annotations_test.go b/pkg/protocols/http/request_annotations_test.go
index c558de6e22..aefb22d84f 100644
--- a/pkg/protocols/http/request_annotations_test.go
+++ b/pkg/protocols/http/request_annotations_test.go
@@ -5,7 +5,15 @@ import (
"net/http"
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/http/httpclientpool"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/request_annotations_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http/httpclientpool"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/httpclientpool"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/request_annotations_test.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
"github.com/stretchr/testify/require"
)
diff --git a/pkg/protocols/http/request_fuzz.go b/pkg/protocols/http/request_fuzz.go
index 53ab0f26cf..a77ae233f5 100644
--- a/pkg/protocols/http/request_fuzz.go
+++ b/pkg/protocols/http/request_fuzz.go
@@ -23,6 +23,21 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz"
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz/analyzers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/useragent"
urlutil "github.com/projectdiscovery/utils/url"
@@ -113,6 +128,7 @@ func (request *Request) executeFuzzingRule(input *contextargs.Context, previous
// executeAllFuzzingRules executes all fuzzing rules defined in template for a given base request
func (request *Request) executeAllFuzzingRules(input *contextargs.Context, values map[string]interface{}, baseRequest *retryablehttp.Request, callback protocols.OutputEventCallback) error {
applicable := false
+ values = generators.MergeMaps(request.filterDataMap(input), values)
for _, rule := range request.Fuzzing {
select {
case <-input.Context().Done():
@@ -120,8 +136,9 @@ func (request *Request) executeAllFuzzingRules(input *contextargs.Context, value
default:
}
- err := rule.Execute(&fuzz.ExecuteRuleInput{
- Input: input,
+ input := &fuzz.ExecuteRuleInput{
+ Input: input,
+ DisplayFuzzPoints: request.options.Options.DisplayFuzzPoints,
Callback: func(gr fuzz.GeneratedRequest) bool {
select {
case <-input.Context().Done():
@@ -134,12 +151,19 @@ func (request *Request) executeAllFuzzingRules(input *contextargs.Context, value
},
Values: values,
BaseRequest: baseRequest.Clone(context.TODO()),
- })
+ }
+ if request.Analyzer != nil {
+ analyzer := analyzers.GetAnalyzer(request.Analyzer.Name)
+ input.ApplyPayloadInitialTransformation = analyzer.ApplyInitialTransformation
+ input.AnalyzerParams = request.Analyzer.Parameters
+ }
+ err := rule.Execute(input)
if err == nil {
applicable = true
continue
}
if fuzz.IsErrRuleNotApplicable(err) {
+ gologger.Verbose().Msgf("[%s] fuzz: rule not applicable : %s\n", request.options.TemplateID, err)
continue
}
if err == types.ErrNoMoreRequests {
@@ -158,15 +182,16 @@ func (request *Request) executeAllFuzzingRules(input *contextargs.Context, value
func (request *Request) executeGeneratedFuzzingRequest(gr fuzz.GeneratedRequest, input *contextargs.Context, callback protocols.OutputEventCallback) bool {
hasInteractMatchers := interactsh.HasMatchers(request.CompiledOperators)
hasInteractMarkers := len(gr.InteractURLs) > 0
- if request.options.HostErrorsCache != nil && request.options.HostErrorsCache.Check(input.MetaInput.Input) {
+ if request.options.HostErrorsCache != nil && request.options.HostErrorsCache.Check(request.options.ProtocolType.String(), input) {
return false
}
request.options.RateLimitTake()
req := &generatedRequest{
- request: gr.Request,
- dynamicValues: gr.DynamicValues,
- interactshURLs: gr.InteractURLs,
- original: request,
+ request: gr.Request,
+ dynamicValues: gr.DynamicValues,
+ interactshURLs: gr.InteractURLs,
+ original: request,
+ fuzzGeneratedRequest: gr,
}
var gotMatches bool
requestErr := request.executeRequest(input, req, gr.DynamicValues, hasInteractMatchers, func(event *output.InternalWrappedEvent) {
@@ -177,6 +202,7 @@ func (request *Request) executeGeneratedFuzzingRequest(gr fuzz.GeneratedRequest,
result.FuzzingPosition = gr.Component.Name()
}
+ setInteractshCallback := false
if hasInteractMarkers && hasInteractMatchers && request.options.Interactsh != nil {
requestData := &interactsh.RequestData{
MakeResultFunc: request.MakeResultEvent,
@@ -184,7 +210,10 @@ func (request *Request) executeGeneratedFuzzingRequest(gr fuzz.GeneratedRequest,
Operators: request.CompiledOperators,
MatchFunc: request.Match,
ExtractFunc: request.Extract,
+ Parameter: gr.Parameter,
+ Request: gr.Request,
}
+ setInteractshCallback = true
request.options.Interactsh.RequestEvent(gr.InteractURLs, requestData)
gotMatches = request.options.Interactsh.AlreadyMatched(requestData)
} else {
@@ -194,6 +223,13 @@ func (request *Request) executeGeneratedFuzzingRequest(gr fuzz.GeneratedRequest,
if event.OperatorsResult != nil {
gotMatches = event.OperatorsResult.Matched
}
+ if request.options.FuzzParamsFrequency != nil && !setInteractshCallback {
+ if !gotMatches {
+ request.options.FuzzParamsFrequency.MarkParameter(gr.Parameter, gr.Request.URL.String(), request.options.TemplateID)
+ } else {
+ request.options.FuzzParamsFrequency.UnmarkParameter(gr.Parameter, gr.Request.URL.String(), request.options.TemplateID)
+ }
+ }
}, 0)
// If a variable is unresolved, skip all further requests
if errors.Is(requestErr, ErrMissingVars) {
@@ -201,7 +237,7 @@ func (request *Request) executeGeneratedFuzzingRequest(gr fuzz.GeneratedRequest,
}
if requestErr != nil {
if request.options.HostErrorsCache != nil {
- request.options.HostErrorsCache.MarkFailed(input.MetaInput.Input, requestErr)
+ request.options.HostErrorsCache.MarkFailed(request.options.ProtocolType.String(), input, requestErr)
}
gologger.Verbose().Msgf("[%s] Error occurred in request: %s\n", request.options.TemplateID, requestErr)
}
@@ -222,7 +258,12 @@ func (request *Request) ShouldFuzzTarget(input *contextargs.Context) bool {
}
status := []bool{}
for index, filter := range request.FuzzPreCondition {
- isMatch, _ := request.Match(request.filterDataMap(input), filter)
+ dataMap := request.filterDataMap(input)
+ // dump if svd is enabled
+ if request.options.Options.ShowVarDump {
+ gologger.Debug().Msgf("Fuzz Filter Variables: \n%s\n", vardump.DumpVariables(dataMap))
+ }
+ isMatch, _ := request.Match(dataMap, filter)
status = append(status, isMatch)
if request.options.Options.MatcherStatus {
gologger.Debug().Msgf("[%s] [%s] Filter => %s : %v", input.MetaInput.Target(), request.options.TemplateID, operators.GetMatcherName(filter, index), isMatch)
@@ -283,10 +324,5 @@ func (request *Request) filterDataMap(input *contextargs.Context) map[string]int
// add default method value
m["method"] = http.MethodGet
}
-
- // dump if svd is enabled
- if request.options.Options.ShowVarDump {
- gologger.Debug().Msgf("Fuzz Filter Variables: \n%s\n", vardump.DumpVariables(m))
- }
return m
}
diff --git a/pkg/protocols/http/request_generator.go b/pkg/protocols/http/request_generator.go
index 9418041eb4..40e6ef3aca 100644
--- a/pkg/protocols/http/request_generator.go
+++ b/pkg/protocols/http/request_generator.go
@@ -1,8 +1,18 @@
package http
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/request_generator.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/request_generator.go
+>>>>>>> projectdiscovery-main
)
// requestGenerator generates requests sequentially based on various
diff --git a/pkg/protocols/http/request_generator_test.go b/pkg/protocols/http/request_generator_test.go
index 45eb78877d..af6542eafa 100644
--- a/pkg/protocols/http/request_generator_test.go
+++ b/pkg/protocols/http/request_generator_test.go
@@ -5,9 +5,20 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/disk"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/request_generator_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/disk"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/request_generator_test.go
+>>>>>>> projectdiscovery-main
)
func TestRequestGeneratorPaths(t *testing.T) {
diff --git a/pkg/protocols/http/signature.go b/pkg/protocols/http/signature.go
index 96ce201a6b..cde43fb5db 100644
--- a/pkg/protocols/http/signature.go
+++ b/pkg/protocols/http/signature.go
@@ -7,7 +7,15 @@ import (
"github.com/invopop/jsonschema"
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/http/signer"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/signature.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http/signer"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/signer"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/signature.go
+>>>>>>> projectdiscovery-main
)
// SignatureType is the type of signature
diff --git a/pkg/protocols/http/signer/signer.go b/pkg/protocols/http/signer/signer.go
index 7f6e01fa30..039d29c90e 100644
--- a/pkg/protocols/http/signer/signer.go
+++ b/pkg/protocols/http/signer/signer.go
@@ -5,7 +5,15 @@ import (
"errors"
"net/http"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/signer/signer.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/signer/signer.go
+>>>>>>> projectdiscovery-main
)
// An Argument that can be passed to Signer
diff --git a/pkg/protocols/http/signerpool/signerpool.go b/pkg/protocols/http/signerpool/signerpool.go
index 741a157ca1..1e3ae811e9 100644
--- a/pkg/protocols/http/signerpool/signerpool.go
+++ b/pkg/protocols/http/signerpool/signerpool.go
@@ -5,9 +5,21 @@ import (
"strings"
"sync"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/protocols/http/signer"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/http/signerpool/signerpool.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http/signer"
+
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http/signer"
+
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/http/signerpool/signerpool.go
+>>>>>>> projectdiscovery-main
)
var (
diff --git a/pkg/protocols/javascript/js.go b/pkg/protocols/javascript/js.go
index 2a30ab24bd..17847db0df 100644
--- a/pkg/protocols/javascript/js.go
+++ b/pkg/protocols/javascript/js.go
@@ -32,8 +32,31 @@ import (
"github.com/dop251/goja"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/js/compiler"
+ "github.com/projectdiscovery/nuclei/v3/pkg/js/gojs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/utils/errkit"
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
iputil "github.com/projectdiscovery/utils/ip"
+ mapsutil "github.com/projectdiscovery/utils/maps"
syncutil "github.com/projectdiscovery/utils/sync"
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -64,9 +87,6 @@ type Request struct {
// Code contains code to execute for the javascript request.
Code string `yaml:"code,omitempty" json:"code,omitempty" jsonschema:"title=code to execute in javascript,description=Executes inline javascript code for the request"`
// description: |
- // Timeout in seconds is optional timeout for each javascript script execution (i.e init, pre-condition, code)
- Timeout int `yaml:"timeout,omitempty" json:"timeout,omitempty" jsonschema:"title=timeout for javascript execution,description=Timeout in seconds is optional timeout for entire javascript script execution"`
- // description: |
// StopAtFirstMatch stops processing the request at first match.
StopAtFirstMatch bool `yaml:"stop-at-first-match,omitempty" json:"stop-at-first-match,omitempty" jsonschema:"title=stop at first match,description=Stop the execution after a match is found"`
// description: |
@@ -152,9 +172,9 @@ func (request *Request) Compile(options *protocols.ExecutorOptions) error {
}
opts := &compiler.ExecuteOptions{
- Timeout: request.Timeout,
- Source: &request.Init,
- Context: context.Background(),
+ TimeoutVariants: request.options.Options.GetTimeouts(),
+ Source: &request.Init,
+ Context: context.Background(),
}
// register 'export' function to export variables from init code
// these are saved in args and are available in pre-condition and request code
@@ -320,7 +340,7 @@ func (request *Request) ExecuteWithResults(target *contextargs.Context, dynamicV
templateCtx.Merge(payloadValues)
if vardump.EnableVarDump {
- gologger.Debug().Msgf("Javascript Protocol request variables: \n%s\n", vardump.DumpVariables(payloadValues))
+ gologger.Debug().Msgf("JavaScript Protocol request variables: %s\n", vardump.DumpVariables(payloadValues))
}
if request.PreCondition != "" {
@@ -344,23 +364,42 @@ func (request *Request) ExecuteWithResults(target *contextargs.Context, dynamicV
argsCopy.TemplateCtx = templateCtx.GetAll()
result, err := request.options.JsCompiler.ExecuteWithOptions(request.preConditionCompiled, argsCopy,
- &compiler.ExecuteOptions{Timeout: request.Timeout, Source: &request.PreCondition, Context: target.Context()})
- if err != nil {
- return errorutil.NewWithTag(request.TemplateID, "could not execute pre-condition: %s", err)
- }
- if !result.GetSuccess() || types.ToString(result["error"]) != "" {
- gologger.Warning().Msgf("[%s] Precondition for request %s was not satisfied\n", request.TemplateID, request.PreCondition)
- request.options.Progress.IncrementFailedRequestsBy(1)
- return nil
- }
- if request.options.Options.Debug || request.options.Options.DebugRequests {
- request.options.Progress.IncrementRequests()
- gologger.Debug().Msgf("[%s] Precondition for request was satisfied\n", request.TemplateID)
+ &compiler.ExecuteOptions{
+ TimeoutVariants: requestOptions.Options.GetTimeouts(),
+ Source: &request.PreCondition, Context: target.Context(),
+ })
+ // if precondition was successful
+ if err == nil && result.GetSuccess() {
+ if request.options.Options.Debug || request.options.Options.DebugRequests {
+ request.options.Progress.IncrementRequests()
+ gologger.Debug().Msgf("[%s] Precondition for request was satisfied\n", request.TemplateID)
+ }
+ } else {
+ var outError error
+ // if js code failed to execute
+ if err != nil {
+ outError = errkit.Append(errkit.New("pre-condition not satisfied skipping template execution"), err)
+ } else {
+ // execution successful but pre-condition returned false
+ outError = errkit.New("pre-condition not satisfied skipping template execution")
+ }
+ results := map[string]interface{}(result)
+ results["error"] = outError.Error()
+ // generate and return failed event
+ data := request.generateEventData(input, results, hostPort)
+ data = generators.MergeMaps(data, payloadValues)
+ event := eventcreator.CreateEventWithAdditionalOptions(request, data, request.options.Options.Debug || request.options.Options.DebugResponse, func(wrappedEvent *output.InternalWrappedEvent) {
+ allVars := argsCopy.Map()
+ allVars = generators.MergeMaps(allVars, data)
+ wrappedEvent.OperatorsResult.PayloadValues = allVars
+ })
+ callback(event)
+ return err
}
}
if request.generator != nil && request.Threads > 1 {
- request.executeRequestParallel(context.Background(), hostPort, hostname, input, payloadValues, callback)
+ request.executeRequestParallel(target.Context(), hostPort, hostname, input, payloadValues, callback)
return nil
}
@@ -387,11 +426,10 @@ func (request *Request) ExecuteWithResults(target *contextargs.Context, dynamicV
}
callback(result)
}, requestOptions); err != nil {
- _ = err
- // Review: should we log error here?
- // it is technically not error as it is expected to fail
- // gologger.Warning().Msgf("Could not execute request: %s\n", err)
- // do not return even if error occured
+ if errkit.IsNetworkPermanentErr(err) {
+ // gologger.Verbose().Msgf("Could not execute request: %s\n", err)
+ return err
+ }
}
// If this was a match, and we want to stop at first match, skip all further requests.
shouldStopAtFirstMatch := request.options.Options.StopAtFirstMatch || request.StopAtFirstMatch
@@ -408,8 +446,8 @@ func (request *Request) executeRequestParallel(ctxParent context.Context, hostPo
if threads == 0 {
threads = 1
}
- ctx, cancel := context.WithCancel(ctxParent)
- defer cancel()
+ ctx, cancel := context.WithCancelCause(ctxParent)
+ defer cancel(nil)
requestOptions := request.options
gotmatches := &atomic.Bool{}
@@ -453,16 +491,15 @@ func (request *Request) executeRequestParallel(ctxParent context.Context, hostPo
}
callback(result)
}, requestOptions); err != nil {
- _ = err
- // Review: should we log error here?
- // it is technically not error as it is expected to fail
- // gologger.Warning().Msgf("Could not execute request: %s\n", err)
- // do not return even if error occured
+ if errkit.IsNetworkPermanentErr(err) {
+ cancel(err)
+ return
+ }
}
// If this was a match, and we want to stop at first match, skip all further requests.
if shouldStopAtFirstMatch && gotmatches.Load() {
- cancel()
+ cancel(nil)
return
}
}()
@@ -501,13 +538,16 @@ func (request *Request) executeRequestWithPayloads(hostPort string, input *conte
}
results, err := request.options.JsCompiler.ExecuteWithOptions(request.scriptCompiled, argsCopy,
- &compiler.ExecuteOptions{Timeout: request.Timeout, Source: &request.Code, Context: input.Context()})
+ &compiler.ExecuteOptions{
+ TimeoutVariants: requestOptions.Options.GetTimeouts(),
+ Source: &request.Code,
+ Context: input.Context(),
+ })
if err != nil {
// shouldn't fail even if it returned error instead create a failure event
results = compiler.ExecuteResult{"success": false, "error": err.Error()}
}
request.options.Progress.IncrementRequests()
-
requestOptions.Output.Request(requestOptions.TemplateID, hostPort, request.Type().String(), err)
gologger.Verbose().Msgf("[%s] Sent Javascript request to %s", request.options.TemplateID, hostPort)
@@ -529,24 +569,72 @@ func (request *Request) executeRequestWithPayloads(hostPort string, input *conte
}
}
+ values := mapsutil.Merge(payloadValues, results)
+ // generate event data
+ data := request.generateEventData(input, values, hostPort)
+
+ // add and get values from templatectx
+ request.options.AddTemplateVars(input.MetaInput, request.Type(), request.GetID(), data)
+ data = generators.MergeMaps(data, request.options.GetTemplateCtx(input.MetaInput).GetAll())
+
+ if requestOptions.Options.Debug || requestOptions.Options.DebugRequests || requestOptions.Options.StoreResponse {
+ msg := fmt.Sprintf("[%s] Dumped Javascript response for %s:\n%v", requestOptions.TemplateID, input.MetaInput.Input, vardump.DumpVariables(results))
+ if requestOptions.Options.Debug || requestOptions.Options.DebugRequests {
+ gologger.Debug().Str("address", input.MetaInput.Input).Msg(msg)
+ }
+ if requestOptions.Options.StoreResponse {
+ request.options.Output.WriteStoreDebugData(input.MetaInput.Input, request.options.TemplateID, request.Type().String(), msg)
+ }
+ }
+
+ if _, ok := data["error"]; ok {
+ event := eventcreator.CreateEventWithAdditionalOptions(request, generators.MergeMaps(data, payloadValues), request.options.Options.Debug || request.options.Options.DebugResponse, func(wrappedEvent *output.InternalWrappedEvent) {
+ wrappedEvent.OperatorsResult.PayloadValues = payload
+ })
+ callback(event)
+ return err
+ }
+
+ if request.options.Interactsh != nil {
+ request.options.Interactsh.MakePlaceholders(interactshURLs, data)
+ }
+
+ var event *output.InternalWrappedEvent
+ if len(interactshURLs) == 0 {
+ event = eventcreator.CreateEventWithAdditionalOptions(request, generators.MergeMaps(data, payloadValues), request.options.Options.Debug || request.options.Options.DebugResponse, func(wrappedEvent *output.InternalWrappedEvent) {
+ wrappedEvent.OperatorsResult.PayloadValues = payload
+ })
+ callback(event)
+ } else if request.options.Interactsh != nil {
+ event = &output.InternalWrappedEvent{InternalEvent: data, UsesInteractsh: true}
+ request.options.Interactsh.RequestEvent(interactshURLs, &interactsh.RequestData{
+ MakeResultFunc: request.MakeResultEvent,
+ Event: event,
+ Operators: request.CompiledOperators,
+ MatchFunc: request.Match,
+ ExtractFunc: request.Extract,
+ })
+ }
+ return nil
+}
+
+// generateEventData generates event data for the request
+func (request *Request) generateEventData(input *contextargs.Context, values map[string]interface{}, matched string) map[string]interface{} {
data := make(map[string]interface{})
- for k, v := range payloadValues {
+ for k, v := range values {
data[k] = v
}
data["type"] = request.Type().String()
- for k, v := range results {
- data[k] = v
- }
+ data["request-pre-condition"] = beautifyJavascript(request.PreCondition)
data["request"] = beautifyJavascript(request.Code)
data["host"] = input.MetaInput.Input
- data["matched"] = hostPort
- data["template-path"] = requestOptions.TemplatePath
- data["template-id"] = requestOptions.TemplateID
- data["template-info"] = requestOptions.TemplateInfo
+ data["matched"] = matched
+ data["template-path"] = request.options.TemplatePath
+ data["template-id"] = request.options.TemplateID
+ data["template-info"] = request.options.TemplateInfo
if request.StopAtFirstMatch || request.options.StopAtFirstMatch {
data["stop-at-first-match"] = true
}
-
// add ip address to data
if input.MetaInput.CustomIP != "" {
data["ip"] = input.MetaInput.CustomIP
@@ -586,50 +674,7 @@ func (request *Request) executeRequestWithPayloads(hostPort string, input *conte
}
}
}
-
- // add and get values from templatectx
- request.options.AddTemplateVars(input.MetaInput, request.Type(), request.GetID(), data)
- data = generators.MergeMaps(data, request.options.GetTemplateCtx(input.MetaInput).GetAll())
-
- if requestOptions.Options.Debug || requestOptions.Options.DebugRequests || requestOptions.Options.StoreResponse {
- msg := fmt.Sprintf("[%s] Dumped Javascript response for %s:\n%v", requestOptions.TemplateID, input.MetaInput.Input, vardump.DumpVariables(results))
- if requestOptions.Options.Debug || requestOptions.Options.DebugRequests {
- gologger.Debug().Str("address", input.MetaInput.Input).Msg(msg)
- }
- if requestOptions.Options.StoreResponse {
- request.options.Output.WriteStoreDebugData(input.MetaInput.Input, request.options.TemplateID, request.Type().String(), msg)
- }
- }
-
- if _, ok := data["error"]; ok {
- event := eventcreator.CreateEventWithAdditionalOptions(request, generators.MergeMaps(data, payloadValues), request.options.Options.Debug || request.options.Options.DebugResponse, func(wrappedEvent *output.InternalWrappedEvent) {
- wrappedEvent.OperatorsResult.PayloadValues = payload
- })
- callback(event)
- return err
- }
-
- if request.options.Interactsh != nil {
- request.options.Interactsh.MakePlaceholders(interactshURLs, data)
- }
-
- var event *output.InternalWrappedEvent
- if len(interactshURLs) == 0 {
- event = eventcreator.CreateEventWithAdditionalOptions(request, generators.MergeMaps(data, payloadValues), request.options.Options.Debug || request.options.Options.DebugResponse, func(wrappedEvent *output.InternalWrappedEvent) {
- wrappedEvent.OperatorsResult.PayloadValues = payload
- })
- callback(event)
- } else if request.options.Interactsh != nil {
- event = &output.InternalWrappedEvent{InternalEvent: data, UsesInteractsh: true}
- request.options.Interactsh.RequestEvent(interactshURLs, &interactsh.RequestData{
- MakeResultFunc: request.MakeResultEvent,
- Event: event,
- Operators: request.CompiledOperators,
- MatchFunc: request.Match,
- ExtractFunc: request.Extract,
- })
- }
- return nil
+ return data
}
func (request *Request) getArgsCopy(input *contextargs.Context, payloadValues map[string]interface{}, requestOptions *protocols.ExecutorOptions, ignoreErrors bool) (*compiler.ExecuteArgs, error) {
@@ -742,6 +787,7 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(wrapped.InternalEvent["template-id"]),
TemplatePath: types.ToString(wrapped.InternalEvent["template-path"]),
Info: wrapped.InternalEvent["template-info"].(model.Info),
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Host: fields.Host,
Port: fields.Port,
diff --git a/pkg/protocols/network/network.go b/pkg/protocols/network/network.go
index aed275e3fc..dfdcdb9332 100644
--- a/pkg/protocols/network/network.go
+++ b/pkg/protocols/network/network.go
@@ -12,7 +12,24 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/network/networkclientpool"
"github.com/projectdiscovery/fastdialer/fastdialer"
+<<<<<<< HEAD
errorutil "github.com/projectdiscovery/utils/errors"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/network/network.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/network/networkclientpool"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/network/networkclientpool"
+ errorutil "github.com/projectdiscovery/utils/errors"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/network/network.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
)
@@ -85,6 +102,10 @@ type Request struct {
// SelfContained specifies if the request is self-contained.
SelfContained bool `yaml:"-" json:"-"`
+ // description: |
+ // StopAtFirstMatch stops the execution of the requests and template as soon as a match is found.
+ StopAtFirstMatch bool `yaml:"stop-at-first-match,omitempty" json:"stop-at-first-match,omitempty" jsonschema:"title=stop at first match,description=Stop the execution after a match is found"`
+
// description: |
// ports is post processed list of ports to scan (obtained from Port)
ports []string `yaml:"-" json:"-"`
diff --git a/pkg/protocols/network/network_test.go b/pkg/protocols/network/network_test.go
index bc19a11452..8780fe9887 100644
--- a/pkg/protocols/network/network_test.go
+++ b/pkg/protocols/network/network_test.go
@@ -5,9 +5,21 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/network/network_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/network/network_test.go
+>>>>>>> projectdiscovery-main
)
func TestNetworkCompileMake(t *testing.T) {
diff --git a/pkg/protocols/network/networkclientpool/clientpool.go b/pkg/protocols/network/networkclientpool/clientpool.go
index e8d968cac8..da41c793b0 100644
--- a/pkg/protocols/network/networkclientpool/clientpool.go
+++ b/pkg/protocols/network/networkclientpool/clientpool.go
@@ -4,6 +4,16 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/fastdialer/fastdialer"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/network/networkclientpool/clientpool.go
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolstate"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/network/networkclientpool/clientpool.go
+>>>>>>> projectdiscovery-main
)
var (
diff --git a/pkg/protocols/network/operators.go b/pkg/protocols/network/operators.go
index fd1bf06baa..1be527e69f 100644
--- a/pkg/protocols/network/operators.go
+++ b/pkg/protocols/network/operators.go
@@ -3,6 +3,7 @@ package network
import (
"time"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/operators/extractors"
@@ -11,6 +12,26 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
protocolutils "github.com/Explorer1092/nuclei/v3/pkg/protocols/utils"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/network/operators.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/network/operators.go
+>>>>>>> projectdiscovery-main
)
// Match matches a generic data response again a given matcher
@@ -103,6 +124,10 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(wrapped.InternalEvent["template-id"]),
TemplatePath: types.ToString(wrapped.InternalEvent["template-path"]),
Info: wrapped.InternalEvent["template-info"].(model.Info),
+<<<<<<< HEAD
+=======
+ TemplateVerifier: request.options.TemplateVerifier,
+>>>>>>> projectdiscovery-main
Type: types.ToString(wrapped.InternalEvent["type"]),
Host: fields.Host,
Port: fields.Port,
diff --git a/pkg/protocols/network/operators_test.go b/pkg/protocols/network/operators_test.go
index f2a0c5e39d..37ac96b1b0 100644
--- a/pkg/protocols/network/operators_test.go
+++ b/pkg/protocols/network/operators_test.go
@@ -5,6 +5,7 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
@@ -12,6 +13,25 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/operators/matchers"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/network/operators_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/network/operators_test.go
+>>>>>>> projectdiscovery-main
)
func TestResponseToDSLMap(t *testing.T) {
diff --git a/pkg/protocols/network/request.go b/pkg/protocols/network/request.go
index df700b4c8c..76657dbda4 100644
--- a/pkg/protocols/network/request.go
+++ b/pkg/protocols/network/request.go
@@ -1,7 +1,6 @@
package network
import (
- "context"
"encoding/hex"
"fmt"
"net"
@@ -9,6 +8,7 @@ import (
"os"
"strings"
"sync"
+ "sync/atomic"
"time"
"github.com/pkg/errors"
@@ -30,6 +30,23 @@ import (
protocolutils "github.com/Explorer1092/nuclei/v3/pkg/protocols/utils"
templateTypes "github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/replacer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
mapsutil "github.com/projectdiscovery/utils/maps"
"github.com/projectdiscovery/utils/reader"
@@ -64,7 +81,7 @@ func (request *Request) getOpenPorts(target *contextargs.Context) ([]string, err
errs = append(errs, err)
continue
}
- conn, err := protocolstate.Dialer.Dial(context.TODO(), "tcp", addr)
+ conn, err := protocolstate.Dialer.Dial(target.Context(), "tcp", addr)
if err != nil {
errs = append(errs, err)
continue
@@ -100,6 +117,16 @@ func (request *Request) ExecuteWithResults(target *contextargs.Context, metadata
gologger.Verbose().Msgf("[%v] got errors while checking open ports: %s\n", request.options.TemplateID, err)
}
+ // stop at first match if requested
+ atomicBool := &atomic.Bool{}
+ shouldStopAtFirstMatch := request.StopAtFirstMatch || request.options.StopAtFirstMatch || request.options.Options.StopAtFirstMatch
+ wrappedCallback := func(event *output.InternalWrappedEvent) {
+ if event != nil && event.HasOperatorResult() {
+ atomicBool.Store(true)
+ }
+ callback(event)
+ }
+
for _, port := range ports {
input := target.Clone()
// use network port updates input with new port requested in template file
@@ -108,9 +135,12 @@ func (request *Request) ExecuteWithResults(target *contextargs.Context, metadata
if err := input.UseNetworkPort(port, request.ExcludePorts); err != nil {
gologger.Debug().Msgf("Could not network port from constants: %s\n", err)
}
- if err := request.executeOnTarget(input, visitedAddresses, metadata, previous, callback); err != nil {
+ if err := request.executeOnTarget(input, visitedAddresses, metadata, previous, wrappedCallback); err != nil {
return err
}
+ if shouldStopAtFirstMatch && atomicBool.Load() {
+ break
+ }
}
return nil
@@ -119,6 +149,10 @@ func (request *Request) ExecuteWithResults(target *contextargs.Context, metadata
func (request *Request) executeOnTarget(input *contextargs.Context, visited mapsutil.Map[string, struct{}], metadata, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
var address string
var err error
+ if request.isUnresponsiveAddress(input) {
+ // skip on unresponsive address no need to continue
+ return nil
+ }
if request.SelfContained {
address = ""
@@ -138,6 +172,16 @@ func (request *Request) executeOnTarget(input *contextargs.Context, visited maps
variablesMap := request.options.Variables.Evaluate(variables)
variables = generators.MergeMaps(variablesMap, variables, request.options.Constants)
+ // stop at first match if requested
+ atomicBool := &atomic.Bool{}
+ shouldStopAtFirstMatch := request.StopAtFirstMatch || request.options.StopAtFirstMatch || request.options.Options.StopAtFirstMatch
+ wrappedCallback := func(event *output.InternalWrappedEvent) {
+ if event != nil && event.HasOperatorResult() {
+ atomicBool.Store(true)
+ }
+ callback(event)
+ }
+
for _, kv := range request.addresses {
select {
case <-input.Context().Done():
@@ -151,15 +195,16 @@ func (request *Request) executeOnTarget(input *contextargs.Context, visited maps
continue
}
visited.Set(actualAddress, struct{}{})
-
- if err := request.executeAddress(variables, actualAddress, address, input, kv.tls, previous, callback); err != nil {
+ if err = request.executeAddress(variables, actualAddress, address, input, kv.tls, previous, wrappedCallback); err != nil {
outputEvent := request.responseToDSLMap("", "", "", address, "")
callback(&output.InternalWrappedEvent{InternalEvent: outputEvent})
gologger.Warning().Msgf("[%v] Could not make network request for (%s) : %s\n", request.options.TemplateID, actualAddress, err)
- continue
+ }
+ if shouldStopAtFirstMatch && atomicBool.Load() {
+ break
}
}
- return nil
+ return err
}
// executeAddress executes the request for an address
@@ -173,6 +218,8 @@ func (request *Request) executeAddress(variables map[string]interface{}, actualA
request.options.Progress.IncrementFailedRequestsBy(1)
return err
}
+ updatedTarget := input.Clone()
+ updatedTarget.MetaInput.Input = actualAddress
// if request threads matches global payload concurrency we follow it
shouldFollowGlobal := request.Threads == request.options.Options.PayloadConcurrency
@@ -206,11 +253,19 @@ func (request *Request) executeAddress(variables map[string]interface{}, actualA
m.Unlock()
}
}
+ if request.isUnresponsiveAddress(updatedTarget) {
+ // skip on unresponsive address no need to continue
+ return nil
+ }
value = generators.MergeMaps(value, payloads)
swg.Add()
go func(vars map[string]interface{}) {
defer swg.Done()
+ if request.isUnresponsiveAddress(updatedTarget) {
+ // skip on unresponsive address no need to continue
+ return
+ }
if err := request.executeRequestWithPayloads(variables, actualAddress, address, input, shouldUseTLS, vars, previous, callback); err != nil {
m.Lock()
multiErr = multierr.Append(multiErr, err)
@@ -240,13 +295,22 @@ func (request *Request) executeRequestWithPayloads(variables map[string]interfac
if host, _, err := net.SplitHostPort(actualAddress); err == nil {
hostname = host
}
+ updatedTarget := input.Clone()
+ updatedTarget.MetaInput.Input = actualAddress
+
+ if request.isUnresponsiveAddress(updatedTarget) {
+ // skip on unresponsive address no need to continue
+ return nil
+ }
if shouldUseTLS {
- conn, err = request.dialer.DialTLS(context.Background(), "tcp", actualAddress)
+ conn, err = request.dialer.DialTLS(input.Context(), "tcp", actualAddress)
} else {
- conn, err = request.dialer.Dial(context.Background(), "tcp", actualAddress)
+ conn, err = request.dialer.Dial(input.Context(), "tcp", actualAddress)
}
if err != nil {
+ // adds it to unresponsive address list if applicable
+ request.markUnresponsiveAddress(updatedTarget, err)
request.options.Output.Request(request.options.TemplatePath, address, request.Type().String(), err)
request.options.Progress.IncrementFailedRequestsBy(1)
return errors.Wrap(err, "could not connect to server")
@@ -261,36 +325,37 @@ func (request *Request) executeRequestWithPayloads(variables map[string]interfac
interimValues := generators.MergeMaps(variables, payloads)
if vardump.EnableVarDump {
- gologger.Debug().Msgf("Network Protocol request variables: \n%s\n", vardump.DumpVariables(interimValues))
+ gologger.Debug().Msgf("Network Protocol request variables: %s\n", vardump.DumpVariables(interimValues))
}
inputEvents := make(map[string]interface{})
for _, input := range request.Inputs {
- data := []byte(input.Data)
-
- if request.options.Interactsh != nil {
- var transformedData string
- transformedData, interactshURLs = request.options.Interactsh.Replace(string(data), []string{})
- data = []byte(transformedData)
- }
+ dataInBytes := []byte(input.Data)
+ var err error
- finalData, err := expressions.EvaluateByte(data, interimValues)
+ dataInBytes, err = expressions.EvaluateByte(dataInBytes, interimValues)
if err != nil {
request.options.Output.Request(request.options.TemplatePath, address, request.Type().String(), err)
request.options.Progress.IncrementFailedRequestsBy(1)
return errors.Wrap(err, "could not evaluate template expressions")
}
- reqBuilder.Write(finalData)
+ data := string(dataInBytes)
+ if request.options.Interactsh != nil {
+ data, interactshURLs = request.options.Interactsh.Replace(data, []string{})
+ dataInBytes = []byte(data)
+ }
+
+ reqBuilder.Write(dataInBytes)
- if err := expressions.ContainsUnresolvedVariables(string(finalData)); err != nil {
+ if err := expressions.ContainsUnresolvedVariables(data); err != nil {
gologger.Warning().Msgf("[%s] Could not make network request for %s: %v\n", request.options.TemplateID, actualAddress, err)
return nil
}
if input.Type.GetType() == hexType {
- finalData, err = hex.DecodeString(string(finalData))
+ dataInBytes, err = hex.DecodeString(data)
if err != nil {
request.options.Output.Request(request.options.TemplatePath, address, request.Type().String(), err)
request.options.Progress.IncrementFailedRequestsBy(1)
@@ -298,14 +363,14 @@ func (request *Request) executeRequestWithPayloads(variables map[string]interfac
}
}
- if _, err := conn.Write(finalData); err != nil {
+ if _, err := conn.Write(dataInBytes); err != nil {
request.options.Output.Request(request.options.TemplatePath, address, request.Type().String(), err)
request.options.Progress.IncrementFailedRequestsBy(1)
return errors.Wrap(err, "could not write request to server")
}
if input.Read > 0 {
- buffer, err := ConnReadNWithTimeout(conn, int64(input.Read), request.options.Options.ResponseReadTimeout)
+ buffer, err := ConnReadNWithTimeout(conn, int64(input.Read), request.options.Options.GetTimeouts().TcpReadTimeout)
if err != nil {
return errorutil.NewWithErr(err).Msgf("could not read response from connection")
}
@@ -355,7 +420,7 @@ func (request *Request) executeRequestWithPayloads(variables map[string]interfac
bufferSize = -1
}
- final, err := ConnReadNWithTimeout(conn, int64(bufferSize), request.options.Options.ResponseReadTimeout)
+ final, err := ConnReadNWithTimeout(conn, int64(bufferSize), request.options.Options.GetTimeouts().TcpReadTimeout)
if err != nil {
request.options.Output.Request(request.options.TemplatePath, address, request.Type().String(), err)
gologger.Verbose().Msgf("could not read more data from %s: %s", actualAddress, err)
@@ -475,3 +540,21 @@ func ConnReadNWithTimeout(conn net.Conn, n int64, timeout time.Duration) ([]byte
}
return b[:count], nil
}
+
+// markUnresponsiveAddress checks if the error is a unreponsive host error and marks it
+func (request *Request) markUnresponsiveAddress(input *contextargs.Context, err error) {
+ if err == nil {
+ return
+ }
+ if request.options.HostErrorsCache != nil {
+ request.options.HostErrorsCache.MarkFailed(request.options.ProtocolType.String(), input, err)
+ }
+}
+
+// isUnresponsiveAddress checks if the error is a unreponsive based on its execution history
+func (request *Request) isUnresponsiveAddress(input *contextargs.Context) bool {
+ if request.options.HostErrorsCache != nil {
+ return request.options.HostErrorsCache.Check(request.options.ProtocolType.String(), input)
+ }
+ return false
+}
diff --git a/pkg/protocols/network/request_test.go b/pkg/protocols/network/request_test.go
index 0ef84fedd7..726b7496a9 100644
--- a/pkg/protocols/network/request_test.go
+++ b/pkg/protocols/network/request_test.go
@@ -11,6 +11,7 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
@@ -19,6 +20,27 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/network/request_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/network/request_test.go
+>>>>>>> projectdiscovery-main
)
func TestNetworkExecuteWithResults(t *testing.T) {
@@ -86,7 +108,7 @@ func TestNetworkExecuteWithResults(t *testing.T) {
err := request.ExecuteWithResults(ctxArgs, metadata, previous, func(event *output.InternalWrappedEvent) {
finalEvent = event
})
- require.Nil(t, err, "could not execute network request")
+ require.NotNil(t, err, "could not execute network request")
})
require.Nil(t, finalEvent.Results, "could not get event output from request")
diff --git a/pkg/protocols/offlinehttp/find_test.go b/pkg/protocols/offlinehttp/find_test.go
index 058f99ba0f..192e2b6623 100644
--- a/pkg/protocols/offlinehttp/find_test.go
+++ b/pkg/protocols/offlinehttp/find_test.go
@@ -7,11 +7,26 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
permissionutil "github.com/projectdiscovery/utils/permission"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/offlinehttp/find_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+ permissionutil "github.com/projectdiscovery/utils/permission"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/offlinehttp/find_test.go
+>>>>>>> projectdiscovery-main
)
func TestFindResponses(t *testing.T) {
diff --git a/pkg/protocols/offlinehttp/offlinehttp.go b/pkg/protocols/offlinehttp/offlinehttp.go
index 869ead20fd..96a937f64d 100644
--- a/pkg/protocols/offlinehttp/offlinehttp.go
+++ b/pkg/protocols/offlinehttp/offlinehttp.go
@@ -3,8 +3,18 @@ package offlinehttp
import (
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/offlinehttp/offlinehttp.go
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/offlinehttp/offlinehttp.go
+>>>>>>> projectdiscovery-main
)
// Request is a offline http response processing request
diff --git a/pkg/protocols/offlinehttp/operators.go b/pkg/protocols/offlinehttp/operators.go
index 3b656d5dca..4fe8476180 100644
--- a/pkg/protocols/offlinehttp/operators.go
+++ b/pkg/protocols/offlinehttp/operators.go
@@ -5,6 +5,7 @@ import (
"strings"
"time"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/operators/extractors"
@@ -14,6 +15,29 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/utils"
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/offlinehttp/operators.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/utils"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/offlinehttp/operators.go
+>>>>>>> projectdiscovery-main
)
// Match matches a generic data response again a given matcher
@@ -142,6 +166,7 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(wrapped.InternalEvent["template-id"]),
TemplatePath: types.ToString(wrapped.InternalEvent["template-path"]),
Info: wrapped.InternalEvent["template-info"].(model.Info),
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Path: types.ToString(wrapped.InternalEvent["path"]),
Matched: types.ToString(wrapped.InternalEvent["matched"]),
diff --git a/pkg/protocols/offlinehttp/operators_test.go b/pkg/protocols/offlinehttp/operators_test.go
index 9f2b088a78..654795586c 100644
--- a/pkg/protocols/offlinehttp/operators_test.go
+++ b/pkg/protocols/offlinehttp/operators_test.go
@@ -7,6 +7,7 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
@@ -14,6 +15,25 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/operators/matchers"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/offlinehttp/operators_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/offlinehttp/operators_test.go
+>>>>>>> projectdiscovery-main
)
func TestResponseToDSLMap(t *testing.T) {
diff --git a/pkg/protocols/offlinehttp/request.go b/pkg/protocols/offlinehttp/request.go
index 7b950faaf6..c85f668817 100644
--- a/pkg/protocols/offlinehttp/request.go
+++ b/pkg/protocols/offlinehttp/request.go
@@ -15,13 +15,45 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/utils"
templateTypes "github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
"github.com/projectdiscovery/utils/conversion"
syncutil "github.com/projectdiscovery/utils/sync"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/offlinehttp/request.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/tostring"
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/output"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/tostring"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/utils/conversion"
+ syncutil "github.com/projectdiscovery/utils/sync"
+ unitutils "github.com/projectdiscovery/utils/unit"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/offlinehttp/request.go
+>>>>>>> projectdiscovery-main
)
var _ protocols.Request = &Request{}
-const maxSize = 5 * 1024 * 1024
+const maxSize = 5 * unitutils.Mega
// Type returns the type of the protocol request
func (request *Request) Type() templateTypes.ProtocolType {
diff --git a/pkg/protocols/protocols.go b/pkg/protocols/protocols.go
index 4045cb6c1b..2356cd988d 100644
--- a/pkg/protocols/protocols.go
+++ b/pkg/protocols/protocols.go
@@ -11,6 +11,7 @@ import (
"github.com/logrusorgru/aurora"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/authprovider"
"github.com/Explorer1092/nuclei/v3/pkg/catalog"
"github.com/Explorer1092/nuclei/v3/pkg/input"
@@ -37,6 +38,37 @@ import (
var (
MaxTemplateFileSizeForEncoding = 1024 * 1024
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog"
+ "github.com/projectdiscovery/nuclei/v3/pkg/fuzz/frequency"
+ "github.com/projectdiscovery/nuclei/v3/pkg/input"
+ "github.com/projectdiscovery/nuclei/v3/pkg/js/compiler"
+ "github.com/projectdiscovery/nuclei/v3/pkg/loader/parser"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/progress"
+ "github.com/projectdiscovery/nuclei/v3/pkg/projectfile"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/globalmatchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hosterrorscache"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/excludematchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/variables"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless/engine"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ unitutils "github.com/projectdiscovery/utils/unit"
+)
+
+var (
+ MaxTemplateFileSizeForEncoding = unitutils.Mega
+>>>>>>> projectdiscovery-main
)
// Executer is an interface implemented any protocol based request executer.
@@ -59,6 +91,8 @@ type ExecutorOptions struct {
TemplatePath string
// TemplateInfo contains information block of the template request
TemplateInfo model.Info
+ // TemplateVerifier is the verifier for the template
+ TemplateVerifier string
// RawTemplate is the raw template for the request
RawTemplate []byte
// Output is a writer interface for writing output events from executer.
@@ -92,6 +126,8 @@ type ExecutorOptions struct {
ExcludeMatchers *excludematchers.ExcludeMatchers
// InputHelper is a helper for input normalization
InputHelper *input.Helper
+ // FuzzParamsFrequency is a cache for parameter frequency
+ FuzzParamsFrequency *frequency.Tracker
Operators []*operators.Operators // only used by offlinehttp module
@@ -120,6 +156,8 @@ type ExecutorOptions struct {
// ExportReqURLPattern exports the request URL pattern
// in ResultEvent it contains the exact url pattern (ex: {{BaseURL}}/{{randstr}}/xyz) used in the request
ExportReqURLPattern bool
+ // GlobalMatchers is the storage for global matchers with http passive templates
+ GlobalMatchers *globalmatchers.Storage
}
// todo: centralizing components is not feasible with current clogged architecture
@@ -193,6 +231,11 @@ func (e *ExecutorOptions) AddTemplateVars(input *contextargs.MetaInput, reqType
}
templateCtx := e.GetTemplateCtx(input)
for k, v := range vars {
+ if stringsutil.HasPrefixAny(k, templateTypes.SupportedProtocolsStrings()...) {
+ // this was inherited from previous protocols no need to modify it we can directly set it or omit
+ templateCtx.Set(k, v)
+ continue
+ }
if !stringsutil.EqualFoldAny(k, "template-id", "template-info", "template-path") {
if reqID != "" {
k = reqID + "_" + k
@@ -212,6 +255,11 @@ func (e *ExecutorOptions) AddTemplateVar(input *contextargs.MetaInput, templateT
return
}
templateCtx := e.GetTemplateCtx(input)
+ if stringsutil.HasPrefixAny(key, templateTypes.SupportedProtocolsStrings()...) {
+ // this was inherited from previous protocols no need to modify it we can directly set it or omit
+ templateCtx.Set(key, value)
+ return
+ }
if reqID != "" {
key = reqID + "_" + key
} else if templateType < templateTypes.InvalidProtocol {
diff --git a/pkg/protocols/ssl/ssl.go b/pkg/protocols/ssl/ssl.go
index 074ec55d5d..af30af2574 100644
--- a/pkg/protocols/ssl/ssl.go
+++ b/pkg/protocols/ssl/ssl.go
@@ -3,8 +3,10 @@ package ssl
import (
"fmt"
"net"
+ "strings"
"time"
+ "github.com/cespare/xxhash"
"github.com/fatih/structs"
jsoniter "github.com/json-iterator/go"
"github.com/pkg/errors"
@@ -27,12 +29,70 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/fastdialer/fastdialer"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/ssl/ssl.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/dns"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/network/networkclientpool"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/utils"
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/model"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v2/pkg/output"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/network/networkclientpool"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ protocolutils "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/network/networkclientpool"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/ssl/ssl.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/tlsx/pkg/tlsx"
"github.com/projectdiscovery/tlsx/pkg/tlsx/clients"
"github.com/projectdiscovery/tlsx/pkg/tlsx/openssl"
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
- urlutil "github.com/projectdiscovery/utils/url"
)
// Request is a request for the SSL protocol
@@ -99,15 +159,15 @@ type Request struct {
options *protocols.ExecutorOptions
}
-// CanCluster returns true if the request can be clustered.
-func (request *Request) CanCluster(other *Request) bool {
- if len(request.CipherSuites) > 0 || request.MinVersion != "" || request.MaxVersion != "" {
- return false
- }
- if request.Address != other.Address || request.ScanMode != other.ScanMode {
- return false
- }
- return true
+// TmplClusterKey generates a unique key for the request
+// to be used in the clustering process.
+func (request *Request) TmplClusterKey() uint64 {
+ inp := fmt.Sprintf("%s-%s-%t-%t-%s", request.Address, request.ScanMode, request.TLSCiphersEnum, request.TLSVersionsEnum, strings.Join(request.TLSCipherTypes, ","))
+ return xxhash.Sum64String(inp)
+}
+
+func (request *Request) IsClusterable() bool {
+ return !(len(request.CipherSuites) > 0 || request.MinVersion != "" || request.MaxVersion != "")
}
// Compile compiles the request generators preparing any requests possible.
@@ -197,10 +257,7 @@ func (request *Request) GetID() string {
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
- hostPort, err := getAddress(input.MetaInput.Input)
- if err != nil {
- return err
- }
+ hostPort := input.MetaInput.Input
hostname, port, _ := net.SplitHostPort(hostPort)
requestOptions := request.options
@@ -224,7 +281,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
payloadValues = generators.MergeMaps(variablesMap, payloadValues, request.options.Constants)
if vardump.EnableVarDump {
- gologger.Debug().Msgf("SSL Protocol request variables: \n%s\n", vardump.DumpVariables(payloadValues))
+ gologger.Debug().Msgf("SSL Protocol request variables: %s\n", vardump.DumpVariables(payloadValues))
}
finalAddress, dataErr := expressions.EvaluateByte([]byte(request.Address), payloadValues)
@@ -349,24 +406,33 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
- "type": "Type is the type of request made",
- "response": "JSON SSL protocol handshake details",
- "not_after": "Timestamp after which the remote cert expires",
- "host": "Host is the input to the template",
- "matched": "Matched is the input which was matched upon",
-}
-
-// getAddress returns the address of the host to make request to
-func getAddress(toTest string) (string, error) {
- urlx, err := urlutil.Parse(toTest)
- if err != nil {
- // use given input instead of url parsing failure
- return toTest, nil
- }
- if urlx.Port() == "" {
- urlx.UpdatePort("443")
- }
- return urlx.Host, nil
+ "template-id": "ID of the template executed",
+ "template-info": "Info Block of the template executed",
+ "template-path": "Path of the template executed",
+ "host": "Host is the input to the template",
+ "port": "Port is the port of the host",
+ "matched": "Matched is the input which was matched upon",
+ "type": "Type is the type of request made",
+ "timestamp": "Timestamp is the time when the request was made",
+ "response": "JSON SSL protocol handshake details",
+ "cipher": "Cipher is the encryption algorithm used",
+ "domains": "Domains are the list of domain names in the certificate",
+ "fingerprint_hash": "Fingerprint hash is the unique identifier of the certificate",
+ "ip": "IP is the IP address of the server",
+ "issuer_cn": "Issuer CN is the common name of the certificate issuer",
+ "issuer_dn": "Issuer DN is the distinguished name of the certificate issuer",
+ "issuer_org": "Issuer organization is the organization of the certificate issuer",
+ "not_after": "Timestamp after which the remote cert expires",
+ "not_before": "Timestamp before which the certificate is not valid",
+ "probe_status": "Probe status indicates if the probe was successful",
+ "serial": "Serial is the serial number of the certificate",
+ "sni": "SNI is the server name indication used in the handshake",
+ "subject_an": "Subject AN is the list of subject alternative names",
+ "subject_cn": "Subject CN is the common name of the certificate subject",
+ "subject_dn": "Subject DN is the distinguished name of the certificate subject",
+ "subject_org": "Subject organization is the organization of the certificate subject",
+ "tls_connection": "TLS connection is the type of TLS connection used",
+ "tls_version": "TLS version is the version of the TLS protocol used",
}
// Match performs matching operation for a matcher on model and returns:
@@ -413,6 +479,7 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(wrapped.InternalEvent["template-id"]),
TemplatePath: types.ToString(wrapped.InternalEvent["template-path"]),
Info: wrapped.InternalEvent["template-info"].(model.Info),
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Host: fields.Host,
Port: fields.Port,
diff --git a/pkg/protocols/ssl/ssl_test.go b/pkg/protocols/ssl/ssl_test.go
index eac0cc79f2..d4aeeda56b 100644
--- a/pkg/protocols/ssl/ssl_test.go
+++ b/pkg/protocols/ssl/ssl_test.go
@@ -6,11 +6,27 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/ssl/ssl_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/ssl/ssl_test.go
+>>>>>>> projectdiscovery-main
)
func TestSSLProtocol(t *testing.T) {
@@ -36,8 +52,3 @@ func TestSSLProtocol(t *testing.T) {
require.Nil(t, err, "could not run ssl request")
require.NotEmpty(t, gotEvent, "could not get event items")
}
-
-func TestGetAddress(t *testing.T) {
- address, _ := getAddress("https://scanme.sh")
- require.Equal(t, "scanme.sh:443", address, "could not get correct address")
-}
diff --git a/pkg/protocols/utils/http/variables.go b/pkg/protocols/utils/http/variables.go
new file mode 100644
index 0000000000..edd12331f7
--- /dev/null
+++ b/pkg/protocols/utils/http/variables.go
@@ -0,0 +1,66 @@
+package utils
+
+import (
+ "fmt"
+ "path"
+ "strings"
+
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/dns"
+ urlutil "github.com/projectdiscovery/utils/url"
+)
+
+// GenerateVariables will create default variables with context args
+func GenerateVariablesWithContextArgs(input *contextargs.Context, trailingSlash bool) map[string]interface{} {
+ parsed, err := urlutil.Parse(input.MetaInput.Input)
+ if err != nil {
+ return nil
+ }
+ return GenerateVariablesWithURL(parsed, trailingSlash, contextargs.GenerateVariables(input))
+}
+
+// GenerateVariables will create default variables after parsing a url with additional variables
+func GenerateVariablesWithURL(inputURL *urlutil.URL, trailingSlash bool, additionalVars map[string]interface{}) map[string]interface{} {
+ parsed := inputURL.Clone()
+ // Query parameter merging is handled elsewhere and should not be included in {{BaseURL}} or other httpVariables
+ parsed.Params = make(urlutil.Params)
+ domain := parsed.Host
+ if strings.Contains(parsed.Host, ":") {
+ domain = strings.Split(parsed.Host, ":")[0]
+ }
+
+ port := parsed.Port()
+ if port == "" {
+ if parsed.Scheme == "https" {
+ port = "443"
+ } else if parsed.Scheme == "http" {
+ port = "80"
+ }
+ }
+
+ if trailingSlash {
+ parsed.Path = strings.TrimSuffix(parsed.Path, "/")
+ }
+
+ escapedPath := parsed.EscapedPath()
+ directory := path.Dir(escapedPath)
+ if directory == "." {
+ directory = ""
+ }
+ base := path.Base(escapedPath)
+ if base == "." {
+ base = ""
+ }
+ httpVariables := map[string]interface{}{
+ "BaseURL": parsed.String(),
+ "RootURL": fmt.Sprintf("%s://%s", parsed.Scheme, parsed.Host),
+ "Hostname": parsed.Host,
+ "Host": domain,
+ "Port": port,
+ "Path": directory,
+ "File": base,
+ "Scheme": parsed.Scheme,
+ }
+ return generators.MergeMaps(httpVariables, dns.GenerateVariables(domain), additionalVars)
+}
diff --git a/pkg/protocols/utils/http/variables_test.go b/pkg/protocols/utils/http/variables_test.go
new file mode 100644
index 0000000000..b820e95dbe
--- /dev/null
+++ b/pkg/protocols/utils/http/variables_test.go
@@ -0,0 +1,63 @@
+package utils
+
+import (
+ "testing"
+
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ urlutil "github.com/projectdiscovery/utils/url"
+ "github.com/stretchr/testify/require"
+)
+
+func TestVariables(t *testing.T) {
+ baseURL := "http://localhost:9001/test/123"
+ parsed, _ := urlutil.Parse(baseURL)
+ // trailingslash is only true when both target/inputURL and payload {{BaseURL}}/xyz both have slash
+ values := GenerateVariablesWithURL(parsed, false, nil)
+
+ require.Equal(t, values["BaseURL"], parsed.String(), "incorrect baseurl")
+ require.Equal(t, values["RootURL"], "http://localhost:9001", "incorrect rootURL")
+ require.Equal(t, values["Host"], "localhost", "incorrect domain name")
+ require.Equal(t, values["Path"], "/test", "incorrect path")
+ require.Equal(t, values["File"], "123", "incorrect file")
+ require.Equal(t, values["Port"], "9001", "incorrect port number")
+ require.Equal(t, values["Scheme"], "http", "incorrect scheme")
+ require.Equal(t, values["Hostname"], "localhost:9001", "incorrect hostname")
+
+ baseURL = "https://example.com"
+ parsed, _ = urlutil.Parse(baseURL)
+ values = GenerateVariablesWithURL(parsed, false, nil)
+
+ require.Equal(t, values["BaseURL"], parsed.String(), "incorrect baseurl")
+ require.Equal(t, values["Host"], "example.com", "incorrect domain name")
+ require.Equal(t, values["RootURL"], "https://example.com", "incorrect rootURL")
+ require.Equal(t, values["Path"], "", "incorrect path")
+ require.Equal(t, values["Port"], "443", "incorrect port number")
+ require.Equal(t, values["Scheme"], "https", "incorrect scheme")
+ require.Equal(t, values["Hostname"], "example.com", "incorrect hostname")
+
+ baseURL = "ftp://foobar.com/"
+ parsed, _ = urlutil.Parse(baseURL)
+ values = GenerateVariablesWithURL(parsed, false, nil)
+
+ require.Equal(t, values["BaseURL"], parsed.String(), "incorrect baseurl")
+ require.Equal(t, values["Host"], "foobar.com", "incorrect domain name")
+ require.Equal(t, values["RootURL"], "ftp://foobar.com", "incorrect rootURL")
+ require.Equal(t, values["Path"], "/", "incorrect path")
+ require.Equal(t, values["Port"], "", "incorrect port number") // Unsupported protocol results in a blank port
+ require.Equal(t, values["Scheme"], "ftp", "incorrect scheme")
+ require.Equal(t, values["Hostname"], "foobar.com", "incorrect hostname")
+
+ baseURL = "http://scanme.sh"
+ ctxArgs := contextargs.NewWithInput(baseURL)
+ ctxArgs.MetaInput.CustomIP = "1.2.3.4"
+ values = GenerateVariablesWithContextArgs(ctxArgs, true)
+
+ require.Equal(t, values["BaseURL"], baseURL, "incorrect baseurl")
+ require.Equal(t, values["Host"], "scanme.sh", "incorrect domain name")
+ require.Equal(t, values["RootURL"], "http://scanme.sh", "incorrect rootURL")
+ require.Equal(t, values["Path"], "", "incorrect path")
+ require.Equal(t, values["Port"], "80", "incorrect port number")
+ require.Equal(t, values["Scheme"], "http", "incorrect scheme")
+ require.Equal(t, values["Hostname"], "scanme.sh", "incorrect hostname")
+ require.Equal(t, values["ip"], "1.2.3.4", "incorrect ip")
+}
diff --git a/pkg/protocols/utils/utils.go b/pkg/protocols/utils/utils.go
index a7ceed4469..e2b17d477a 100644
--- a/pkg/protocols/utils/utils.go
+++ b/pkg/protocols/utils/utils.go
@@ -7,7 +7,15 @@ import (
"os"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/utils/utils.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/utils/utils.go
+>>>>>>> projectdiscovery-main
)
// CleanStructFieldJSONTag cleans struct json tag field
diff --git a/pkg/protocols/websocket/websocket.go b/pkg/protocols/websocket/websocket.go
index 88e79e69ab..4a21a8258d 100644
--- a/pkg/protocols/websocket/websocket.go
+++ b/pkg/protocols/websocket/websocket.go
@@ -1,7 +1,6 @@
package websocket
import (
- "context"
"crypto/tls"
"fmt"
"io"
@@ -33,6 +32,59 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/fastdialer/fastdialer"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/websocket/websocket.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/network/networkclientpool"
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v2/pkg/output"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/network/networkclientpool"
+ protocolutils "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/network/networkclientpool"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/websocket/websocket.go
+>>>>>>> projectdiscovery-main
urlutil "github.com/projectdiscovery/utils/url"
)
@@ -208,7 +260,7 @@ func (request *Request) executeRequestWithPayloads(target *contextargs.Context,
}
if vardump.EnableVarDump {
- gologger.Debug().Msgf("Websocket Protocol request variables: \n%s\n", vardump.DumpVariables(payloadValues))
+ gologger.Debug().Msgf("WebSocket Protocol request variables: %s\n", vardump.DumpVariables(payloadValues))
}
finalAddress, dataErr := expressions.EvaluateByte([]byte(request.Address), payloadValues)
@@ -228,7 +280,7 @@ func (request *Request) executeRequestWithPayloads(target *contextargs.Context,
parsedAddress.Path = path.Join(parsedAddress.Path, parsed.Path)
addressToDial = parsedAddress.String()
- conn, readBuffer, _, err := websocketDialer.Dial(context.Background(), addressToDial)
+ conn, readBuffer, _, err := websocketDialer.Dial(target.Context(), addressToDial)
if err != nil {
requestOptions.Output.Request(requestOptions.TemplateID, input, request.Type().String(), err)
requestOptions.Progress.IncrementFailedRequestsBy(1)
@@ -403,6 +455,7 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(request.options.TemplateID),
TemplatePath: types.ToString(request.options.TemplatePath),
Info: request.options.TemplateInfo,
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Host: fields.Host,
Port: fields.Port,
diff --git a/pkg/protocols/whois/rdapclientpool/clientpool.go b/pkg/protocols/whois/rdapclientpool/clientpool.go
index 6eb259d1a2..0684797df1 100644
--- a/pkg/protocols/whois/rdapclientpool/clientpool.go
+++ b/pkg/protocols/whois/rdapclientpool/clientpool.go
@@ -3,6 +3,14 @@ package rdapclientpool
import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/protocols/whois/rdapclientpool/clientpool.go
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/whois/rdapclientpool/clientpool.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/rdap"
)
diff --git a/pkg/protocols/whois/whois.go b/pkg/protocols/whois/whois.go
index 1651d493c3..a6bab95af6 100644
--- a/pkg/protocols/whois/whois.go
+++ b/pkg/protocols/whois/whois.go
@@ -24,8 +24,64 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/whois/rdapclientpool"
templateTypes "github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+<<<<<<< HEAD:v2/pkg/protocols/whois/whois.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/replacer"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/utils/vardump"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/whois/rdapclientpool"
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+ urlutil "github.com/projectdiscovery/utils/url"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v2/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v2/pkg/output"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/replacer"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/utils/vardump"
+ protocolutils "github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/whois/rdapclientpool"
+ templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
+
+ "github.com/projectdiscovery/nuclei/v2/pkg/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/responsehighlighter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/replacer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
+ protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/whois/rdapclientpool"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/protocols/whois/whois.go
+>>>>>>> projectdiscovery-main
)
// Request is a request for the WHOIS protocol
@@ -99,7 +155,7 @@ func (request *Request) ExecuteWithResults(input *contextargs.Context, dynamicVa
variables := generators.MergeMaps(vars, defaultVars, optionVars, dynamicValues, request.options.Constants)
if vardump.EnableVarDump {
- gologger.Debug().Msgf("Whois Protocol request variables: \n%s\n", vardump.DumpVariables(variables))
+ gologger.Debug().Msgf("Whois Protocol request variables: %s\n", vardump.DumpVariables(variables))
}
// and replace placeholders
@@ -177,6 +233,7 @@ func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent
TemplateID: types.ToString(request.options.TemplateID),
TemplatePath: types.ToString(request.options.TemplatePath),
Info: request.options.TemplateInfo,
+ TemplateVerifier: request.options.TemplateVerifier,
Type: types.ToString(wrapped.InternalEvent["type"]),
Host: types.ToString(wrapped.InternalEvent["host"]),
Metadata: wrapped.OperatorsResult.PayloadValues,
diff --git a/pkg/reporting/client.go b/pkg/reporting/client.go
index 2c54bd1886..6b4890c89a 100644
--- a/pkg/reporting/client.go
+++ b/pkg/reporting/client.go
@@ -1,7 +1,15 @@
package reporting
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/output"
+=======
+<<<<<<< HEAD:v2/pkg/reporting/client.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/client.go
+>>>>>>> projectdiscovery-main
)
// Client is a client for nuclei issue tracking module
diff --git a/pkg/reporting/dedupe/dedupe.go b/pkg/reporting/dedupe/dedupe.go
index 17e6183be5..132630f195 100644
--- a/pkg/reporting/dedupe/dedupe.go
+++ b/pkg/reporting/dedupe/dedupe.go
@@ -11,9 +11,20 @@ import (
"github.com/syndtr/goleveldb/leveldb"
"github.com/syndtr/goleveldb/leveldb/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/conversion"
+=======
+<<<<<<< HEAD:v2/pkg/reporting/dedupe/dedupe.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/utils/conversion"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/dedupe/dedupe.go
+>>>>>>> projectdiscovery-main
)
// Storage is a duplicate detecting storage for nuclei scan events.
diff --git a/pkg/reporting/dedupe/dedupe_test.go b/pkg/reporting/dedupe/dedupe_test.go
index 184b2ce464..f8fda9cb2a 100644
--- a/pkg/reporting/dedupe/dedupe_test.go
+++ b/pkg/reporting/dedupe/dedupe_test.go
@@ -6,7 +6,15 @@ import (
"github.com/stretchr/testify/require"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/output"
+=======
+<<<<<<< HEAD:v2/pkg/reporting/dedupe/dedupe_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/dedupe/dedupe_test.go
+>>>>>>> projectdiscovery-main
)
func TestDedupeDuplicates(t *testing.T) {
diff --git a/pkg/reporting/exporters/es/elasticsearch.go b/pkg/reporting/exporters/es/elasticsearch.go
index 2749f69e6c..73acfcb957 100644
--- a/pkg/reporting/exporters/es/elasticsearch.go
+++ b/pkg/reporting/exporters/es/elasticsearch.go
@@ -12,8 +12,18 @@ import (
"github.com/pkg/errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/protocolstate"
+=======
+<<<<<<< HEAD:v2/pkg/reporting/exporters/es/elasticsearch.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolstate"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/exporters/es/elasticsearch.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/useragent"
)
diff --git a/pkg/reporting/exporters/jsonexporter/jsonexporter.go b/pkg/reporting/exporters/jsonexporter/jsonexporter.go
index e94a442465..956c1bd36f 100644
--- a/pkg/reporting/exporters/jsonexporter/jsonexporter.go
+++ b/pkg/reporting/exporters/jsonexporter/jsonexporter.go
@@ -2,6 +2,14 @@ package jsonexporter
import (
"encoding/json"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/reporting/exporters/jsonexporter/jsonexporter.go
+ "github.com/pkg/errors"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+=======
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/exporters/jsonexporter/jsonexporter.go
+>>>>>>> projectdiscovery-main
"os"
"sync"
diff --git a/pkg/reporting/exporters/jsonl/jsonl.go b/pkg/reporting/exporters/jsonl/jsonl.go
index f791f9d1a5..2bf8824a86 100644
--- a/pkg/reporting/exporters/jsonl/jsonl.go
+++ b/pkg/reporting/exporters/jsonl/jsonl.go
@@ -10,16 +10,21 @@ import (
)
type Exporter struct {
- options *Options
- mutex *sync.Mutex
- rows []output.ResultEvent
+ options *Options
+ mutex *sync.Mutex
+ rows []output.ResultEvent
+ outputFile *os.File
}
// Options contains the configuration options for JSONL exporter client
type Options struct {
// File is the file to export found JSONL result to
- File string `yaml:"file"`
- OmitRaw bool `yaml:"omit-raw"`
+ File string `yaml:"file"`
+ // OmitRaw whether to exclude the raw request and response from the output
+ OmitRaw bool `yaml:"omit-raw"`
+ // BatchSize the number of records to keep in memory before writing them out to the JSONL file or 0 to disable
+ // batching (default)
+ BatchSize int `yaml:"batch-size"`
}
// New creates a new JSONL exporter integration client based on options.
@@ -32,8 +37,7 @@ func New(options *Options) (*Exporter, error) {
return exporter, nil
}
-// Export appends the passed result event to the list of objects to be exported to
-// the resulting JSONL file
+// Export appends the passed result event to the list of objects to be exported to the resulting JSONL file
func (exporter *Exporter) Export(event *output.ResultEvent) error {
exporter.mutex.Lock()
defer exporter.mutex.Unlock()
@@ -46,23 +50,36 @@ func (exporter *Exporter) Export(event *output.ResultEvent) error {
// Add the event to the rows
exporter.rows = append(exporter.rows, *event)
+ // If the batch size is greater than 0 and the number of rows has reached the batch, flush it to the database
+ if exporter.options.BatchSize > 0 && len(exporter.rows) >= exporter.options.BatchSize {
+ err := exporter.WriteRows()
+ if err != nil {
+ // The error is already logged, return it to bubble up to the caller
+ return err
+ }
+ }
+
return nil
}
-// Close writes the in-memory data to the JSONL file specified by options.JSONLExport
-// and closes the exporter after operation
-func (exporter *Exporter) Close() error {
- exporter.mutex.Lock()
- defer exporter.mutex.Unlock()
-
- // Open the JSONL file for writing and create it if it doesn't exist
- f, err := os.OpenFile(exporter.options.File, os.O_WRONLY|os.O_CREATE, 0644)
- if err != nil {
- return errors.Wrap(err, "failed to create JSONL file")
+// WriteRows writes all rows from the rows list to JSONL file and removes them from the list
+func (exporter *Exporter) WriteRows() error {
+ // Open the file for writing if it's not already.
+ // This will recreate the file if it exists, but keep the file handle so that batched writes within the same
+ // execution are appended to the same file.
+ var err error
+ if exporter.outputFile == nil {
+ // Open the JSONL file for writing and create it if it doesn't exist
+ exporter.outputFile, err = os.OpenFile(exporter.options.File, os.O_WRONLY|os.O_CREATE, 0644)
+ if err != nil {
+ return errors.Wrap(err, "failed to create JSONL file")
+ }
}
- // Loop through the rows and convert each to a JSON byte array and write to file
- for _, row := range exporter.rows {
+ // Loop through the rows and write them, removing them as they're entered
+ for len(exporter.rows) > 0 {
+ row := exporter.rows[0]
+
// Convert the row to JSON byte array and append a trailing newline. This is treated as a single line in JSONL
obj, err := json.Marshal(row)
if err != nil {
@@ -70,16 +87,36 @@ func (exporter *Exporter) Close() error {
}
// Add a trailing newline to the JSON byte array to confirm with the JSONL format
- obj = append(obj, '\n')
+ obj = append(obj, ',', '\n')
// Attempt to append the JSON line to file specified in options.JSONLExport
- if _, err = f.Write(obj); err != nil {
+ if _, err = exporter.outputFile.Write(obj); err != nil {
return errors.Wrap(err, "failed to append JSONL line")
}
+
+ // Remove the item from the list
+ exporter.rows = exporter.rows[1:]
+ }
+
+ return nil
+}
+
+// Close writes the in-memory data to the JSONL file specified by options.JSONLExport and closes the exporter after
+// operation
+func (exporter *Exporter) Close() error {
+ exporter.mutex.Lock()
+ defer exporter.mutex.Unlock()
+
+ // Write any remaining rows to the file
+ // Write all pending rows
+ err := exporter.WriteRows()
+ if err != nil {
+ // The error is already logged, return it to bubble up to the caller
+ return err
}
// Close the file
- if err := f.Close(); err != nil {
+ if err := exporter.outputFile.Close(); err != nil {
return errors.Wrap(err, "failed to close JSONL file")
}
diff --git a/pkg/reporting/exporters/markdown/markdown.go b/pkg/reporting/exporters/markdown/markdown.go
index a3569173c3..e01e41f9d8 100644
--- a/pkg/reporting/exporters/markdown/markdown.go
+++ b/pkg/reporting/exporters/markdown/markdown.go
@@ -6,12 +6,29 @@ import (
"path/filepath"
"strings"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/reporting/exporters/markdown/markdown.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/format"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/output"
+ "github.com/projectdiscovery/nuclei/v2/pkg/reporting/exporters/markdown/util"
+ "github.com/projectdiscovery/nuclei/v2/pkg/reporting/format"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/gologger"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/exporters/markdown/util"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/format"
fileutil "github.com/projectdiscovery/utils/file"
+<<<<<<< HEAD
+=======
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/exporters/markdown/markdown.go
+>>>>>>> projectdiscovery-main
stringsutil "github.com/projectdiscovery/utils/strings"
)
diff --git a/pkg/reporting/exporters/mongo/mongo.go b/pkg/reporting/exporters/mongo/mongo.go
new file mode 100644
index 0000000000..faf8bb579b
--- /dev/null
+++ b/pkg/reporting/exporters/mongo/mongo.go
@@ -0,0 +1,155 @@
+package mongo
+
+import (
+ "context"
+ "github.com/pkg/errors"
+ "github.com/projectdiscovery/gologger"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "go.mongodb.org/mongo-driver/mongo"
+ "net/url"
+ "os"
+ "strings"
+ "sync"
+
+ mongooptions "go.mongodb.org/mongo-driver/mongo/options"
+)
+
+type Exporter struct {
+ options *Options
+ mutex *sync.Mutex
+ rows []output.ResultEvent
+ collection *mongo.Collection
+ connection *mongo.Client
+}
+
+// Options contains the configuration options for MongoDB exporter client
+type Options struct {
+ // ConnectionString is the connection string to the MongoDB database
+ ConnectionString string `yaml:"connection-string"`
+ // CollectionName is the name of the MongoDB collection in which to store the results
+ CollectionName string `yaml:"collection-name"`
+ // OmitRaw excludes the Request and Response from the results (helps with filesize)
+ OmitRaw bool `yaml:"omit-raw"`
+ // BatchSize determines the number of results to be kept in memory before writing it to the database or 0 to
+ // persist all in memory and write all results at the end (default)
+ BatchSize int `yaml:"batch-size"`
+}
+
+// New creates a new MongoDB exporter integration client based on options.
+func New(options *Options) (*Exporter, error) {
+ exporter := &Exporter{
+ mutex: &sync.Mutex{},
+ options: options,
+ rows: []output.ResultEvent{},
+ }
+
+ // If the environment variable for the connection string is set, then use that instead. This allows for easier
+ // management of sensitive items such as credentials
+ envConnectionString := os.Getenv("MONGO_CONNECTION_STRING")
+ if envConnectionString != "" {
+ options.ConnectionString = envConnectionString
+ gologger.Info().Msgf("Using connection string from environment variable MONGO_CONNECTION_STRING")
+ }
+
+ // Create the connection to the database
+ clientOptions := mongooptions.Client().ApplyURI(options.ConnectionString)
+
+ // Create a new client and connect to the MongoDB server
+ client, err := mongo.Connect(context.TODO(), clientOptions)
+ if err != nil {
+ gologger.Error().Msgf("Error creating MongoDB client: %s", err)
+ return nil, err
+ }
+
+ // Ensure the connection is valid
+ err = client.Ping(context.Background(), nil)
+ if err != nil {
+ gologger.Error().Msgf("Error connecting to MongoDB: %s", err)
+ return nil, err
+ }
+
+ // Get the database from the connection string to set the database and collection
+ parsed, err := url.Parse(options.ConnectionString)
+ if err != nil {
+ gologger.Error().Msgf("Error parsing connection string: %s", options.ConnectionString)
+ return nil, err
+ }
+
+ databaseName := strings.TrimPrefix(parsed.Path, "/")
+
+ if databaseName == "" {
+ return nil, errors.New("error getting database name from connection string")
+ }
+
+ exporter.connection = client
+ exporter.collection = client.Database(databaseName).Collection(options.CollectionName)
+
+ return exporter, nil
+}
+
+// Export writes a result document to the configured MongoDB collection
+// in the database configured by the connection string
+func (exporter *Exporter) Export(event *output.ResultEvent) error {
+ exporter.mutex.Lock()
+ defer exporter.mutex.Unlock()
+
+ if exporter.options.OmitRaw {
+ event.Request = ""
+ event.Response = ""
+ }
+
+ // Add the row to the queue to be processed
+ exporter.rows = append(exporter.rows, *event)
+
+ // If the batch size is greater than 0 and the number of rows has reached the batch, flush it to the database
+ if exporter.options.BatchSize > 0 && len(exporter.rows) >= exporter.options.BatchSize {
+ err := exporter.WriteRows()
+ if err != nil {
+ // The error is already logged, return it to bubble up to the caller
+ return err
+ }
+ }
+
+ return nil
+}
+
+// WriteRows writes all rows from the rows list to the MongoDB collection and removes them from the list
+func (exporter *Exporter) WriteRows() error {
+ // Loop through the rows and write them, removing them as they're entered
+ for len(exporter.rows) > 0 {
+ data := exporter.rows[0]
+
+ // Write the data to the database
+ _, err := exporter.collection.InsertOne(context.TODO(), data)
+ if err != nil {
+ gologger.Fatal().Msgf("Error inserting record into MongoDB collection: %s", err)
+ return err
+ }
+
+ // Remove the item from the list
+ exporter.rows = exporter.rows[1:]
+ }
+
+ return nil
+}
+
+func (exporter *Exporter) Close() error {
+ exporter.mutex.Lock()
+ defer exporter.mutex.Unlock()
+
+ // Write all pending rows
+ err := exporter.WriteRows()
+ if err != nil {
+ // The error is already logged, return it to bubble up to the caller
+ return err
+ }
+
+ // Close the database connection
+ err = exporter.connection.Disconnect(context.TODO())
+ if err != nil {
+ gologger.Error().Msgf("Error disconnecting from MongoDB: %s", err)
+ return err
+ }
+
+ return nil
+}
diff --git a/pkg/reporting/exporters/sarif/sarif.go b/pkg/reporting/exporters/sarif/sarif.go
index 8b0a130ace..3ca7c50859 100644
--- a/pkg/reporting/exporters/sarif/sarif.go
+++ b/pkg/reporting/exporters/sarif/sarif.go
@@ -10,6 +10,16 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/pkg/errors"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/reporting/exporters/sarif/sarif.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/exporters/sarif/sarif.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/sarif"
)
diff --git a/pkg/reporting/exporters/splunk/splunkhec.go b/pkg/reporting/exporters/splunk/splunkhec.go
index b5dcdf5142..29f63aa754 100644
--- a/pkg/reporting/exporters/splunk/splunkhec.go
+++ b/pkg/reporting/exporters/splunk/splunkhec.go
@@ -13,6 +13,16 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/pkg/errors"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/reporting/exporters/splunk/splunkhec.go
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolstate"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/exporters/splunk/splunkhec.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
"github.com/projectdiscovery/useragent"
)
diff --git a/pkg/reporting/format/format_utils.go b/pkg/reporting/format/format_utils.go
index 648c083bfa..3c6d336c5f 100644
--- a/pkg/reporting/format/format_utils.go
+++ b/pkg/reporting/format/format_utils.go
@@ -6,12 +6,22 @@ import (
"strconv"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/exporters/markdown/util"
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/Explorer1092/nuclei/v3/pkg/utils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/markdown/util"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+ unitutils "github.com/projectdiscovery/utils/unit"
+>>>>>>> projectdiscovery-main
)
// Summary returns a formatted built one line summary of the event
@@ -71,7 +81,7 @@ func CreateReportDescription(event *output.ResultEvent, formatter ResultFormatte
if event.Response != "" {
var responseString string
// If the response is larger than 5 kb, truncate it before writing.
- maxKbSize := 5 * 1024
+ maxKbSize := 5 * unitutils.Kilo
if len(event.Response) > maxKbSize {
responseString = event.Response[:maxKbSize]
responseString += ".... Truncated ...."
@@ -82,7 +92,7 @@ func CreateReportDescription(event *output.ResultEvent, formatter ResultFormatte
}
}
- if len(event.ExtractedResults) > 0 || len(event.Metadata) > 0 {
+ if len(event.ExtractedResults) > 0 || len(event.Metadata) > 0 || event.AnalyzerDetails != "" {
builder.WriteString("\n")
builder.WriteString(formatter.MakeBold("Extra Information"))
builder.WriteString("\n\n")
@@ -98,6 +108,13 @@ func CreateReportDescription(event *output.ResultEvent, formatter ResultFormatte
}
builder.WriteString("\n")
}
+ if event.AnalyzerDetails != "" {
+ builder.WriteString(formatter.MakeBold("Analyzer Details:"))
+ builder.WriteString("\n\n")
+
+ builder.WriteString(event.AnalyzerDetails)
+ builder.WriteString("\n")
+ }
if len(event.Metadata) > 0 {
builder.WriteString(formatter.MakeBold("Metadata:"))
builder.WriteString("\n\n")
diff --git a/pkg/reporting/format/format_utils_test.go b/pkg/reporting/format/format_utils_test.go
index 674602dc0b..5c0f7d2355 100644
--- a/pkg/reporting/format/format_utils_test.go
+++ b/pkg/reporting/format/format_utils_test.go
@@ -4,11 +4,34 @@ import (
"strings"
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/exporters/markdown/util"
"github.com/stretchr/testify/require"
+=======
+<<<<<<< HEAD:v2/pkg/reporting/format/format_utils_test.go
+<<<<<<< HEAD:v2/pkg/reporting/format/format_test.go
+ "github.com/stretchr/testify/assert"
+
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/model"
+ "github.com/projectdiscovery/nuclei/v2/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v2/pkg/reporting/exporters/markdown/util"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93:v2/pkg/reporting/format/format_utils_test.go
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/markdown/util"
+ "github.com/stretchr/testify/require"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/format/format_utils_test.go
+>>>>>>> projectdiscovery-main
)
func TestToMarkdownTableString(t *testing.T) {
diff --git a/pkg/reporting/options.go b/pkg/reporting/options.go
index e01041a138..39d30d1636 100644
--- a/pkg/reporting/options.go
+++ b/pkg/reporting/options.go
@@ -1,6 +1,7 @@
package reporting
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/reporting/exporters/es"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/exporters/jsonexporter"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/exporters/jsonl"
@@ -12,6 +13,33 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/reporting/trackers/github"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/trackers/gitlab"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/trackers/jira"
+=======
+<<<<<<< HEAD:v2/pkg/reporting/options.go
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/es"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/jsonexporter"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/jsonl"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/markdown"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/sarif"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/splunk"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/trackers/github"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/trackers/gitlab"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/trackers/jira"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/es"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonexporter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonl"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/markdown"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/mongo"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/sarif"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/splunk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/filters"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/gitea"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/github"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/gitlab"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/jira"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/linear"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/options.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
)
@@ -29,6 +57,8 @@ type Options struct {
Gitea *gitea.Options `yaml:"gitea"`
// Jira contains configuration options for Jira Issue Tracker
Jira *jira.Options `yaml:"jira"`
+ // Linear contains configuration options for Linear Issue Tracker
+ Linear *linear.Options `yaml:"linear"`
// MarkdownExporter contains configuration options for Markdown Exporter Module
MarkdownExporter *markdown.Options `yaml:"markdown"`
// SarifExporter contains configuration options for Sarif Exporter Module
@@ -41,6 +71,8 @@ type Options struct {
JSONExporter *jsonexporter.Options `yaml:"json"`
// JSONLExporter contains configuration options for JSONL Exporter Module
JSONLExporter *jsonl.Options `yaml:"jsonl"`
+ // MongoDBExporter containers the configuration options for the MongoDB Exporter Module
+ MongoDBExporter *mongo.Options `yaml:"mongodb"`
HttpClient *retryablehttp.Client `yaml:"-"`
OmitRaw bool `yaml:"-"`
diff --git a/pkg/reporting/reporting.go b/pkg/reporting/reporting.go
index aa3f01305a..e1a903088e 100644
--- a/pkg/reporting/reporting.go
+++ b/pkg/reporting/reporting.go
@@ -2,20 +2,35 @@ package reporting
import (
"fmt"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/mongo"
"os"
"strings"
"sync/atomic"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
json_exporter "github.com/Explorer1092/nuclei/v3/pkg/reporting/exporters/jsonexporter"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/exporters/jsonl"
"github.com/projectdiscovery/gologger"
+=======
+<<<<<<< HEAD:v2/pkg/reporting/reporting.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ json_exporter "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/jsonexporter"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/jsonl"
+=======
+ "github.com/projectdiscovery/gologger"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ json_exporter "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonexporter"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/jsonl"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/reporting.go
+>>>>>>> projectdiscovery-main
"go.uber.org/multierr"
"gopkg.in/yaml.v2"
"errors"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/dedupe"
@@ -28,6 +43,35 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/reporting/trackers/github"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/trackers/gitlab"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/trackers/jira"
+=======
+<<<<<<< HEAD:v2/pkg/reporting/reporting.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/dedupe"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/es"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/markdown"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/sarif"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/exporters/splunk"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/trackers/github"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/trackers/gitlab"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/trackers/jira"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/dedupe"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/es"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/markdown"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/sarif"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/splunk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/filters"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/gitea"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/github"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/gitlab"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/jira"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/linear"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/reporting.go
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
)
@@ -112,6 +156,15 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
}
client.trackers = append(client.trackers, tracker)
}
+ if options.Linear != nil {
+ options.Linear.HttpClient = options.HttpClient
+ options.Linear.OmitRaw = options.OmitRaw
+ tracker, err := linear.New(options.Linear)
+ if err != nil {
+ return nil, errorutil.NewWithErr(err).Wrap(ErrReportingClientCreation)
+ }
+ client.trackers = append(client.trackers, tracker)
+ }
if options.MarkdownExporter != nil {
exporter, err := markdown.New(options.MarkdownExporter)
if err != nil {
@@ -156,6 +209,13 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
}
client.exporters = append(client.exporters, exporter)
}
+ if options.MongoDBExporter != nil {
+ exporter, err := mongo.New(options.MongoDBExporter)
+ if err != nil {
+ return nil, errorutil.NewWithErr(err).Wrap(ErrExportClientCreation)
+ }
+ client.exporters = append(client.exporters, exporter)
+ }
if doNotDedupe {
return client, nil
@@ -179,7 +239,7 @@ func New(options *Options, db string, doNotDedupe bool) (Client, error) {
return client, nil
}
-// CreateConfigIfNotExists creates report-config if it doesn't exists
+// CreateConfigIfNotExists creates report-config if it doesn't exist
func CreateConfigIfNotExists() error {
reportingConfig := config.DefaultConfig.GetReportingConfigFilePath()
@@ -195,12 +255,14 @@ func CreateConfigIfNotExists() error {
GitLab: &gitlab.Options{},
Gitea: &gitea.Options{},
Jira: &jira.Options{},
+ Linear: &linear.Options{},
MarkdownExporter: &markdown.Options{},
SarifExporter: &sarif.Options{},
ElasticsearchExporter: &es.Options{},
SplunkExporter: &splunk.Options{},
JSONExporter: &json_exporter.Options{},
JSONLExporter: &jsonl.Options{},
+ MongoDBExporter: &mongo.Options{},
}
reportingFile, err := os.Create(reportingConfig)
if err != nil {
@@ -240,7 +302,7 @@ func (c *ReportingClient) Close() {
if failed > 0 {
msgBuilder.WriteString(fmt.Sprintf(", %d failed", failed))
}
- gologger.Info().Msgf(msgBuilder.String())
+ gologger.Info().Msgf("%v", msgBuilder.String())
}
}
}
@@ -273,7 +335,7 @@ func (c *ReportingClient) CreateIssue(event *output.ResultEvent) error {
for _, tracker := range c.trackers {
// process tracker specific allow/deny list
- if tracker.ShouldFilter(event) {
+ if !tracker.ShouldFilter(event) {
continue
}
diff --git a/pkg/reporting/trackers/gitea/gitea.go b/pkg/reporting/trackers/gitea/gitea.go
index 3635e33f7f..8c6b0fce8a 100644
--- a/pkg/reporting/trackers/gitea/gitea.go
+++ b/pkg/reporting/trackers/gitea/gitea.go
@@ -139,15 +139,15 @@ func (i *Integration) CloseIssue(event *output.ResultEvent) error {
// ShouldFilter determines if an issue should be logged to this tracker
func (i *Integration) ShouldFilter(event *output.ResultEvent) bool {
- if i.options.AllowList != nil && i.options.AllowList.GetMatch(event) {
+ if i.options.AllowList != nil && !i.options.AllowList.GetMatch(event) {
return false
}
if i.options.DenyList != nil && i.options.DenyList.GetMatch(event) {
- return true
+ return false
}
- return false
+ return true
}
func (i *Integration) findIssueByTitle(title string) (*gitea.Issue, error) {
diff --git a/pkg/reporting/trackers/github/github.go b/pkg/reporting/trackers/github/github.go
index 027b0a60f6..888af3444d 100644
--- a/pkg/reporting/trackers/github/github.go
+++ b/pkg/reporting/trackers/github/github.go
@@ -175,15 +175,15 @@ func (i *Integration) Name() string {
// ShouldFilter determines if an issue should be logged to this tracker
func (i *Integration) ShouldFilter(event *output.ResultEvent) bool {
- if i.options.AllowList != nil && i.options.AllowList.GetMatch(event) {
+ if i.options.AllowList != nil && !i.options.AllowList.GetMatch(event) {
return false
}
if i.options.DenyList != nil && i.options.DenyList.GetMatch(event) {
- return true
+ return false
}
- return false
+ return true
}
func (i *Integration) findIssueByTitle(ctx context.Context, title string) (*github.Issue, error) {
diff --git a/pkg/reporting/trackers/gitlab/gitlab.go b/pkg/reporting/trackers/gitlab/gitlab.go
index 05de5837dd..b8008f49f9 100644
--- a/pkg/reporting/trackers/gitlab/gitlab.go
+++ b/pkg/reporting/trackers/gitlab/gitlab.go
@@ -78,7 +78,7 @@ func (i *Integration) CreateIssue(event *output.ResultEvent) (*filters.CreateIss
if label := i.options.IssueLabel; label != "" {
labels = append(labels, label)
}
- customLabels := gitlab.Labels(labels)
+ customLabels := gitlab.LabelOptions(labels)
assigneeIDs := []int{i.userID}
if i.options.DuplicateIssueCheck {
searchIn := "title"
@@ -164,13 +164,13 @@ func (i *Integration) CloseIssue(event *output.ResultEvent) error {
// ShouldFilter determines if an issue should be logged to this tracker
func (i *Integration) ShouldFilter(event *output.ResultEvent) bool {
- if i.options.AllowList != nil && i.options.AllowList.GetMatch(event) {
+ if i.options.AllowList != nil && !i.options.AllowList.GetMatch(event) {
return false
}
if i.options.DenyList != nil && i.options.DenyList.GetMatch(event) {
- return true
+ return false
}
- return false
+ return true
}
diff --git a/pkg/reporting/trackers/jira/jira.go b/pkg/reporting/trackers/jira/jira.go
index 125830457b..70d315c537 100644
--- a/pkg/reporting/trackers/jira/jira.go
+++ b/pkg/reporting/trackers/jira/jira.go
@@ -8,6 +8,7 @@ import (
"sync"
"github.com/andygrunwald/go-jira"
+ "github.com/pkg/errors"
"github.com/trivago/tgo/tcontainer"
"github.com/Explorer1092/nuclei/v3/pkg/output"
@@ -15,7 +16,28 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/reporting/format"
"github.com/Explorer1092/nuclei/v3/pkg/reporting/trackers/filters"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/reporting/trackers/jira/jira.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/reporting/format"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/output"
+ "github.com/projectdiscovery/nuclei/v2/pkg/reporting/exporters/markdown/util"
+ "github.com/projectdiscovery/nuclei/v2/pkg/reporting/format"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/markdown/util"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/format"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/filters"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/reporting/trackers/jira/jira.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
+ "github.com/projectdiscovery/utils/ptr"
)
type Formatter struct {
@@ -69,9 +91,13 @@ type Options struct {
// Token is the token for jira instance.
Token string `yaml:"token" json:"token" validate:"required"`
// ProjectName is the name of the project.
- ProjectName string `yaml:"project-name" json:"project_name" validate:"required"`
+ ProjectName string `yaml:"project-name" json:"project_name"`
+ // ProjectID is the ID of the project (optional)
+ ProjectID string `yaml:"project-id" json:"project_id"`
// IssueType (optional) is the name of the created issue type
IssueType string `yaml:"issue-type" json:"issue_type"`
+ // IssueTypeID (optional) is the ID of the created issue type
+ IssueTypeID string `yaml:"issue-type-id" json:"issue_type_id"`
// SeverityAsLabel (optional) sends the severity as the label of the created
// issue.
SeverityAsLabel bool `yaml:"severity-as-label" json:"severity_as_label"`
@@ -147,13 +173,13 @@ func (i *Integration) CreateNewIssue(event *output.ResultEvent) (*filters.Create
nestedValue = strings.TrimPrefix(fmtNestedValue, "$")
switch nestedValue {
case "CVSSMetrics":
- nestedValue = event.Info.Classification.CVSSMetrics
+ nestedValue = ptr.Safe(event.Info.Classification).CVSSMetrics
case "CVEID":
- nestedValue = event.Info.Classification.CVEID
+ nestedValue = ptr.Safe(event.Info.Classification).CVEID
case "CWEID":
- nestedValue = event.Info.Classification.CWEID
+ nestedValue = ptr.Safe(event.Info.Classification).CWEID
case "CVSSScore":
- nestedValue = event.Info.Classification.CVSSScore
+ nestedValue = ptr.Safe(event.Info.Classification).CVSSScore
case "Host":
nestedValue = event.Host
case "Severity":
@@ -182,6 +208,7 @@ func (i *Integration) CreateNewIssue(event *output.ResultEvent) (*filters.Create
Project: jira.Project{Key: i.options.ProjectName},
Summary: summary,
}
+
// On-prem version of Jira server does not use AccountID
if !i.options.Cloud {
fields = &jira.IssueFields{
@@ -194,6 +221,12 @@ func (i *Integration) CreateNewIssue(event *output.ResultEvent) (*filters.Create
Unknowns: customFields,
}
}
+ if i.options.IssueTypeID != "" {
+ fields.Type = jira.IssueType{ID: i.options.IssueTypeID}
+ }
+ if i.options.ProjectID != "" {
+ fields.Project = jira.Project{ID: i.options.ProjectID}
+ }
issueData := &jira.Issue{
Fields: fields,
@@ -229,18 +262,22 @@ func (i *Integration) CreateIssue(event *output.ResultEvent) (*filters.CreateIss
if i.options.UpdateExisting {
issue, err := i.FindExistingIssue(event)
if err != nil {
- return nil, err
+ return nil, errors.Wrap(err, "could not find existing issue")
} else if issue.ID != "" {
_, _, err = i.jira.Issue.AddComment(issue.ID, &jira.Comment{
Body: format.CreateReportDescription(event, i, i.options.OmitRaw),
})
if err != nil {
- return nil, err
+ return nil, errors.Wrap(err, "could not add comment to existing issue")
}
return getIssueResponseFromJira(&issue)
}
}
- return i.CreateNewIssue(event)
+ resp, err := i.CreateNewIssue(event)
+ if err != nil {
+ return nil, errors.Wrap(err, "could not create new issue")
+ }
+ return resp, nil
}
func (i *Integration) CloseIssue(event *output.ResultEvent) error {
@@ -285,7 +322,11 @@ func (i *Integration) CloseIssue(event *output.ResultEvent) error {
// FindExistingIssue checks if the issue already exists and returns its ID
func (i *Integration) FindExistingIssue(event *output.ResultEvent) (jira.Issue, error) {
template := format.GetMatchedTemplateName(event)
- jql := fmt.Sprintf("summary ~ \"%s\" AND summary ~ \"%s\" AND status != \"%s\" AND project = \"%s\"", template, event.Host, i.options.StatusNot, i.options.ProjectName)
+ project := i.options.ProjectName
+ if i.options.ProjectID != "" {
+ project = i.options.ProjectID
+ }
+ jql := fmt.Sprintf("summary ~ \"%s\" AND summary ~ \"%s\" AND status != \"%s\" AND project = \"%s\"", template, event.Host, i.options.StatusNot, project)
searchOptions := &jira.SearchOptions{
MaxResults: 1, // if any issue exists, then we won't create a new one
@@ -314,13 +355,13 @@ func (i *Integration) FindExistingIssue(event *output.ResultEvent) (jira.Issue,
// ShouldFilter determines if an issue should be logged to this tracker
func (i *Integration) ShouldFilter(event *output.ResultEvent) bool {
- if i.options.AllowList != nil && i.options.AllowList.GetMatch(event) {
+ if i.options.AllowList != nil && !i.options.AllowList.GetMatch(event) {
return false
}
if i.options.DenyList != nil && i.options.DenyList.GetMatch(event) {
- return true
+ return false
}
- return false
+ return true
}
diff --git a/pkg/reporting/trackers/jira/jira_test.go b/pkg/reporting/trackers/jira/jira_test.go
index 247a440c58..d725a97b29 100644
--- a/pkg/reporting/trackers/jira/jira_test.go
+++ b/pkg/reporting/trackers/jira/jira_test.go
@@ -4,6 +4,10 @@ import (
"strings"
"testing"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/filters"
"github.com/stretchr/testify/require"
)
@@ -36,3 +40,33 @@ func TestTableCreation(t *testing.T) {
`
require.Equal(t, expected, table)
}
+
+func Test_ShouldFilter_Tracker(t *testing.T) {
+ jiraIntegration := &Integration{
+ options: &Options{AllowList: &filters.Filter{
+ Severities: severity.Severities{severity.Critical},
+ }},
+ }
+
+ require.False(t, jiraIntegration.ShouldFilter(&output.ResultEvent{Info: model.Info{
+ SeverityHolder: severity.Holder{Severity: severity.Info},
+ }}))
+ require.True(t, jiraIntegration.ShouldFilter(&output.ResultEvent{Info: model.Info{
+ SeverityHolder: severity.Holder{Severity: severity.Critical},
+ }}))
+
+ t.Run("deny-list", func(t *testing.T) {
+ jiraIntegration := &Integration{
+ options: &Options{DenyList: &filters.Filter{
+ Severities: severity.Severities{severity.Critical},
+ }},
+ }
+
+ require.True(t, jiraIntegration.ShouldFilter(&output.ResultEvent{Info: model.Info{
+ SeverityHolder: severity.Holder{Severity: severity.Info},
+ }}))
+ require.False(t, jiraIntegration.ShouldFilter(&output.ResultEvent{Info: model.Info{
+ SeverityHolder: severity.Holder{Severity: severity.Critical},
+ }}))
+ })
+}
diff --git a/pkg/reporting/trackers/linear/jsonutil/jsonutil.go b/pkg/reporting/trackers/linear/jsonutil/jsonutil.go
new file mode 100644
index 0000000000..cf66b7aa1e
--- /dev/null
+++ b/pkg/reporting/trackers/linear/jsonutil/jsonutil.go
@@ -0,0 +1,312 @@
+// Package jsonutil provides a function for decoding JSON
+// into a GraphQL query data structure.
+//
+// Taken from: https://github.com/shurcooL/graphql/blob/ed46e5a4646634fc16cb07c3b8db389542cc8847/internal/jsonutil/graphql.go
+package jsonutil
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "reflect"
+ "strings"
+)
+
+// UnmarshalGraphQL parses the JSON-encoded GraphQL response data and stores
+// the result in the GraphQL query data structure pointed to by v.
+//
+// The implementation is created on top of the JSON tokenizer available
+// in "encoding/json".Decoder.
+func UnmarshalGraphQL(data []byte, v any) error {
+ dec := json.NewDecoder(bytes.NewReader(data))
+ dec.UseNumber()
+ err := (&decoder{tokenizer: dec}).Decode(v)
+ if err != nil {
+ return err
+ }
+ tok, err := dec.Token()
+ switch err {
+ case io.EOF:
+ // Expect to get io.EOF. There shouldn't be any more
+ // tokens left after we've decoded v successfully.
+ return nil
+ case nil:
+ return fmt.Errorf("invalid token '%v' after top-level value", tok)
+ default:
+ return err
+ }
+}
+
+// decoder is a JSON decoder that performs custom unmarshaling behavior
+// for GraphQL query data structures. It's implemented on top of a JSON tokenizer.
+type decoder struct {
+ tokenizer interface {
+ Token() (json.Token, error)
+ }
+
+ // Stack of what part of input JSON we're in the middle of - objects, arrays.
+ parseState []json.Delim
+
+ // Stacks of values where to unmarshal.
+ // The top of each stack is the reflect.Value where to unmarshal next JSON value.
+ //
+ // The reason there's more than one stack is because we might be unmarshaling
+ // a single JSON value into multiple GraphQL fragments or embedded structs, so
+ // we keep track of them all.
+ vs [][]reflect.Value
+}
+
+// Decode decodes a single JSON value from d.tokenizer into v.
+func (d *decoder) Decode(v any) error {
+ rv := reflect.ValueOf(v)
+ if rv.Kind() != reflect.Ptr {
+ return fmt.Errorf("cannot decode into non-pointer %T", v)
+ }
+ d.vs = [][]reflect.Value{{rv.Elem()}}
+ return d.decode()
+}
+
+// decode decodes a single JSON value from d.tokenizer into d.vs.
+func (d *decoder) decode() error {
+ // The loop invariant is that the top of each d.vs stack
+ // is where we try to unmarshal the next JSON value we see.
+ for len(d.vs) > 0 {
+ tok, err := d.tokenizer.Token()
+ if err == io.EOF {
+ return errors.New("unexpected end of JSON input")
+ } else if err != nil {
+ return err
+ }
+
+ switch {
+
+ // Are we inside an object and seeing next key (rather than end of object)?
+ case d.state() == '{' && tok != json.Delim('}'):
+ key, ok := tok.(string)
+ if !ok {
+ return errors.New("unexpected non-key in JSON input")
+ }
+ someFieldExist := false
+ for i := range d.vs {
+ v := d.vs[i][len(d.vs[i])-1]
+ if v.Kind() == reflect.Ptr {
+ v = v.Elem()
+ }
+ var f reflect.Value
+ if v.Kind() == reflect.Struct {
+ f = fieldByGraphQLName(v, key)
+ if f.IsValid() {
+ someFieldExist = true
+ }
+ }
+ d.vs[i] = append(d.vs[i], f)
+ }
+ if !someFieldExist {
+ return fmt.Errorf("struct field for %q doesn't exist in any of %v places to unmarshal", key, len(d.vs))
+ }
+
+ // We've just consumed the current token, which was the key.
+ // Read the next token, which should be the value, and let the rest of code process it.
+ tok, err = d.tokenizer.Token()
+ if err == io.EOF {
+ return errors.New("unexpected end of JSON input")
+ } else if err != nil {
+ return err
+ }
+
+ // Are we inside an array and seeing next value (rather than end of array)?
+ case d.state() == '[' && tok != json.Delim(']'):
+ someSliceExist := false
+ for i := range d.vs {
+ v := d.vs[i][len(d.vs[i])-1]
+ if v.Kind() == reflect.Ptr {
+ v = v.Elem()
+ }
+ var f reflect.Value
+ if v.Kind() == reflect.Slice {
+ v.Set(reflect.Append(v, reflect.Zero(v.Type().Elem()))) // v = append(v, T).
+ f = v.Index(v.Len() - 1)
+ someSliceExist = true
+ }
+ d.vs[i] = append(d.vs[i], f)
+ }
+ if !someSliceExist {
+ return fmt.Errorf("slice doesn't exist in any of %v places to unmarshal", len(d.vs))
+ }
+ }
+
+ switch tok := tok.(type) {
+ case string, json.Number, bool, nil:
+ // Value.
+
+ for i := range d.vs {
+ v := d.vs[i][len(d.vs[i])-1]
+ if !v.IsValid() {
+ continue
+ }
+ err := unmarshalValue(tok, v)
+ if err != nil {
+ return err
+ }
+ }
+ d.popAllVs()
+
+ case json.Delim:
+ switch tok {
+ case '{':
+ // Start of object.
+
+ d.pushState(tok)
+
+ frontier := make([]reflect.Value, len(d.vs)) // Places to look for GraphQL fragments/embedded structs.
+ for i := range d.vs {
+ v := d.vs[i][len(d.vs[i])-1]
+ frontier[i] = v
+ // TODO: Do this recursively or not? Add a test case if needed.
+ if v.Kind() == reflect.Ptr && v.IsNil() {
+ v.Set(reflect.New(v.Type().Elem())) // v = new(T).
+ }
+ }
+ // Find GraphQL fragments/embedded structs recursively, adding to frontier
+ // as new ones are discovered and exploring them further.
+ for len(frontier) > 0 {
+ v := frontier[0]
+ frontier = frontier[1:]
+ if v.Kind() == reflect.Ptr {
+ v = v.Elem()
+ }
+ if v.Kind() != reflect.Struct {
+ continue
+ }
+ for i := 0; i < v.NumField(); i++ {
+ if isGraphQLFragment(v.Type().Field(i)) || v.Type().Field(i).Anonymous {
+ // Add GraphQL fragment or embedded struct.
+ d.vs = append(d.vs, []reflect.Value{v.Field(i)})
+ frontier = append(frontier, v.Field(i))
+ }
+ }
+ }
+ case '[':
+ // Start of array.
+
+ d.pushState(tok)
+
+ for i := range d.vs {
+ v := d.vs[i][len(d.vs[i])-1]
+ // TODO: Confirm this is needed, write a test case.
+ //if v.Kind() == reflect.Ptr && v.IsNil() {
+ // v.Set(reflect.New(v.Type().Elem())) // v = new(T).
+ //}
+
+ // Reset slice to empty (in case it had non-zero initial value).
+ if v.Kind() == reflect.Ptr {
+ v = v.Elem()
+ }
+ if v.Kind() != reflect.Slice {
+ continue
+ }
+ v.Set(reflect.MakeSlice(v.Type(), 0, 0)) // v = make(T, 0, 0).
+ }
+ case '}', ']':
+ // End of object or array.
+ d.popAllVs()
+ d.popState()
+ default:
+ return errors.New("unexpected delimiter in JSON input")
+ }
+ default:
+ return errors.New("unexpected token in JSON input")
+ }
+ }
+ return nil
+}
+
+// pushState pushes a new parse state s onto the stack.
+func (d *decoder) pushState(s json.Delim) {
+ d.parseState = append(d.parseState, s)
+}
+
+// popState pops a parse state (already obtained) off the stack.
+// The stack must be non-empty.
+func (d *decoder) popState() {
+ d.parseState = d.parseState[:len(d.parseState)-1]
+}
+
+// state reports the parse state on top of stack, or 0 if empty.
+func (d *decoder) state() json.Delim {
+ if len(d.parseState) == 0 {
+ return 0
+ }
+ return d.parseState[len(d.parseState)-1]
+}
+
+// popAllVs pops from all d.vs stacks, keeping only non-empty ones.
+func (d *decoder) popAllVs() {
+ var nonEmpty [][]reflect.Value
+ for i := range d.vs {
+ d.vs[i] = d.vs[i][:len(d.vs[i])-1]
+ if len(d.vs[i]) > 0 {
+ nonEmpty = append(nonEmpty, d.vs[i])
+ }
+ }
+ d.vs = nonEmpty
+}
+
+// fieldByGraphQLName returns an exported struct field of struct v
+// that matches GraphQL name, or invalid reflect.Value if none found.
+func fieldByGraphQLName(v reflect.Value, name string) reflect.Value {
+ for i := 0; i < v.NumField(); i++ {
+ if v.Type().Field(i).PkgPath != "" {
+ // Skip unexported field.
+ continue
+ }
+ if hasGraphQLName(v.Type().Field(i), name) {
+ return v.Field(i)
+ }
+ }
+ return reflect.Value{}
+}
+
+// hasGraphQLName reports whether struct field f has GraphQL name.
+func hasGraphQLName(f reflect.StructField, name string) bool {
+ value, ok := f.Tag.Lookup("graphql")
+ if !ok {
+ // TODO: caseconv package is relatively slow. Optimize it, then consider using it here.
+ //return caseconv.MixedCapsToLowerCamelCase(f.Name) == name
+ return strings.EqualFold(f.Name, name)
+ }
+ value = strings.TrimSpace(value) // TODO: Parse better.
+ if strings.HasPrefix(value, "...") {
+ // GraphQL fragment. It doesn't have a name.
+ return false
+ }
+ // Cut off anything that follows the field name,
+ // such as field arguments, aliases, directives.
+ if i := strings.IndexAny(value, "(:@"); i != -1 {
+ value = value[:i]
+ }
+ return strings.TrimSpace(value) == name
+}
+
+// isGraphQLFragment reports whether struct field f is a GraphQL fragment.
+func isGraphQLFragment(f reflect.StructField) bool {
+ value, ok := f.Tag.Lookup("graphql")
+ if !ok {
+ return false
+ }
+ value = strings.TrimSpace(value) // TODO: Parse better.
+ return strings.HasPrefix(value, "...")
+}
+
+// unmarshalValue unmarshals JSON value into v.
+// v must be addressable and not obtained by the use of unexported
+// struct fields, otherwise unmarshalValue will panic.
+func unmarshalValue(value json.Token, v reflect.Value) error {
+ b, err := json.Marshal(value) // TODO: Short-circuit (if profiling says it's worth it).
+ if err != nil {
+ return err
+ }
+ return json.Unmarshal(b, v.Addr().Interface())
+}
diff --git a/pkg/reporting/trackers/linear/linear.go b/pkg/reporting/trackers/linear/linear.go
new file mode 100644
index 0000000000..11712039f8
--- /dev/null
+++ b/pkg/reporting/trackers/linear/linear.go
@@ -0,0 +1,409 @@
+package linear
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "net/http"
+
+ "github.com/shurcooL/graphql"
+
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/exporters/markdown/util"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/format"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/filters"
+ "github.com/projectdiscovery/nuclei/v3/pkg/reporting/trackers/linear/jsonutil"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/retryablehttp-go"
+)
+
+// Integration is a client for linear issue tracker integration
+type Integration struct {
+ url string
+ httpclient *http.Client
+ options *Options
+}
+
+// Options contains the configuration options for linear issue tracker client
+type Options struct {
+ // APIKey is the API key for linear account.
+ APIKey string `yaml:"api-key" validate:"required"`
+
+ // AllowList contains a list of allowed events for this tracker
+ AllowList *filters.Filter `yaml:"allow-list"`
+ // DenyList contains a list of denied events for this tracker
+ DenyList *filters.Filter `yaml:"deny-list"`
+
+ // TeamID is the team id for the project
+ TeamID string `yaml:"team-id"`
+ // ProjectID is the project id for the project
+ ProjectID string `yaml:"project-id"`
+ // DuplicateIssueCheck is a bool to enable duplicate tracking issue check and update the newest
+ DuplicateIssueCheck bool `yaml:"duplicate-issue-check" default:"false"`
+
+ // OpenStateID is the id of the open state for the project
+ OpenStateID string `yaml:"open-state-id"`
+
+ HttpClient *retryablehttp.Client `yaml:"-"`
+ OmitRaw bool `yaml:"-"`
+}
+
+// New creates a new issue tracker integration client based on options.
+func New(options *Options) (*Integration, error) {
+ httpClient := &http.Client{
+ Transport: &addHeaderTransport{
+ T: http.DefaultTransport,
+ Key: options.APIKey,
+ },
+ }
+
+ integration := &Integration{
+ url: "https://api.linear.app/graphql",
+ options: options,
+ httpclient: httpClient,
+ }
+
+ return integration, nil
+}
+
+// CreateIssue creates an issue in the tracker
+func (i *Integration) CreateIssue(event *output.ResultEvent) (*filters.CreateIssueResponse, error) {
+ summary := format.Summary(event)
+ description := format.CreateReportDescription(event, util.MarkdownFormatter{}, i.options.OmitRaw)
+ _ = description
+
+ ctx := context.Background()
+
+ var err error
+ var existingIssue *linearIssue
+ if i.options.DuplicateIssueCheck {
+ existingIssue, err = i.findIssueByTitle(ctx, summary)
+ if err != nil && !errors.Is(err, io.EOF) {
+ return nil, err
+ }
+ }
+
+ if existingIssue == nil {
+ // Create a new issue
+ createdIssue, err := i.createIssueLinear(ctx, summary, description, priorityFromSeverity(event.Info.SeverityHolder.Severity))
+ if err != nil {
+ return nil, err
+ }
+ return &filters.CreateIssueResponse{
+ IssueID: types.ToString(createdIssue.ID),
+ IssueURL: types.ToString(createdIssue.URL),
+ }, nil
+ } else {
+ if existingIssue.State.Name == "Done" {
+ // Update the issue state to open
+ var issueUpdateInput struct {
+ StateID string `json:"stateId"`
+ }
+ issueUpdateInput.StateID = i.options.OpenStateID
+ variables := map[string]interface{}{
+ "issueUpdateInput": issueUpdateInput,
+ "issueID": types.ToString(existingIssue.ID),
+ }
+ var resp struct {
+ IssueUpdate struct {
+ LastSyncID int `json:"lastSyncId"`
+ }
+ }
+ err := i.doGraphqlRequest(ctx, existingIssueUpdateStateMutation, &resp, variables, "IssueUpdate")
+ if err != nil {
+ return nil, fmt.Errorf("error reopening issue %s: %s", existingIssue.ID, err)
+ }
+ }
+
+ commentInput := map[string]interface{}{
+ "issueId": types.ToString(existingIssue.ID),
+ "body": description,
+ }
+ variables := map[string]interface{}{
+ "commentCreateInput": commentInput,
+ }
+ var resp struct {
+ CommentCreate struct {
+ LastSyncID int `json:"lastSyncId"`
+ }
+ }
+ err := i.doGraphqlRequest(ctx, commentCreateExistingTicketMutation, &resp, variables, "CommentCreate")
+ if err != nil {
+ return nil, fmt.Errorf("error commenting on issue %s: %s", existingIssue.ID, err)
+ }
+ return &filters.CreateIssueResponse{
+ IssueID: types.ToString(existingIssue.ID),
+ IssueURL: types.ToString(existingIssue.URL),
+ }, nil
+ }
+}
+
+func priorityFromSeverity(sev severity.Severity) float64 {
+ switch sev {
+ case severity.Critical:
+ return linearPriorityCritical
+ case severity.High:
+ return linearPriorityHigh
+ case severity.Medium:
+ return linearPriorityMedium
+ case severity.Low:
+ return linearPriorityLow
+ default:
+ return linearPriorityNone
+ }
+}
+
+type createIssueMutation struct {
+ IssueCreate struct {
+ Issue struct {
+ ID graphql.ID
+ Title graphql.String
+ Identifier graphql.String
+ State struct {
+ Name graphql.String
+ }
+ URL graphql.String
+ }
+ }
+}
+
+const (
+ createIssueGraphQLMutation = `mutation CreateIssue($input: IssueCreateInput!) {
+ issueCreate(input: $input) {
+ issue {
+ id
+ title
+ identifier
+ state {
+ name
+ }
+ url
+ }
+ }
+}`
+
+ searchExistingTicketQuery = `query ($teamID: ID, $projectID: ID, $title: String!) {
+ issues(filter: {
+ title: { eq: $title },
+ team: { id: { eq: $teamID } }
+ project: { id: { eq: $projectID } }
+ }) {
+ nodes {
+ id
+ title
+ identifier
+ state {
+ name
+ }
+ url
+ }
+ }
+}
+`
+
+ existingIssueUpdateStateMutation = `mutation IssueUpdate($issueUpdateInput: IssueUpdateInput!, $issueID: String!) {
+ issueUpdate(input: $issueUpdateInput, id: $issueID) {
+ lastSyncId
+ }
+}
+`
+
+ commentCreateExistingTicketMutation = `mutation CommentCreate($commentCreateInput: CommentCreateInput!) {
+ commentCreate(input: $commentCreateInput) {
+ lastSyncId
+ }
+}
+`
+)
+
+func (i *Integration) createIssueLinear(ctx context.Context, title, description string, priority float64) (*linearIssue, error) {
+ var mutation createIssueMutation
+ input := map[string]interface{}{
+ "title": title,
+ "description": description,
+ "priority": priority,
+ }
+ if i.options.TeamID != "" {
+ input["teamId"] = graphql.ID(i.options.TeamID)
+ }
+ if i.options.ProjectID != "" {
+ input["projectId"] = i.options.ProjectID
+ }
+
+ variables := map[string]interface{}{
+ "input": input,
+ }
+
+ err := i.doGraphqlRequest(ctx, createIssueGraphQLMutation, &mutation, variables, "CreateIssue")
+ if err != nil {
+ return nil, err
+ }
+
+ return &linearIssue{
+ ID: mutation.IssueCreate.Issue.ID,
+ Title: mutation.IssueCreate.Issue.Title,
+ Identifier: mutation.IssueCreate.Issue.Identifier,
+ State: struct {
+ Name graphql.String
+ }{
+ Name: mutation.IssueCreate.Issue.State.Name,
+ },
+ URL: mutation.IssueCreate.Issue.URL,
+ }, nil
+}
+
+func (i *Integration) findIssueByTitle(ctx context.Context, title string) (*linearIssue, error) {
+ var query findExistingIssuesSearch
+ variables := map[string]interface{}{
+ "title": graphql.String(title),
+ }
+ if i.options.TeamID != "" {
+ variables["teamId"] = graphql.ID(i.options.TeamID)
+ }
+ if i.options.ProjectID != "" {
+ variables["projectID"] = graphql.ID(i.options.ProjectID)
+ }
+
+ err := i.doGraphqlRequest(ctx, searchExistingTicketQuery, &query, variables, "")
+ if err != nil {
+ return nil, err
+ }
+
+ if len(query.Issues.Nodes) > 0 {
+ return &query.Issues.Nodes[0], nil
+ }
+ return nil, io.EOF
+}
+
+func (i *Integration) Name() string {
+ return "linear"
+}
+
+func (i *Integration) CloseIssue(event *output.ResultEvent) error {
+ // TODO: Unimplemented for now as not used in many places
+ // and overhead of maintaining our own API for this.
+ // This is too much code as it is :(
+ return nil
+}
+
+// ShouldFilter determines if an issue should be logged to this tracker
+func (i *Integration) ShouldFilter(event *output.ResultEvent) bool {
+ if i.options.AllowList != nil && !i.options.AllowList.GetMatch(event) {
+ return false
+ }
+
+ if i.options.DenyList != nil && i.options.DenyList.GetMatch(event) {
+ return false
+ }
+
+ return true
+}
+
+type linearIssue struct {
+ ID graphql.ID
+ Title graphql.String
+ Identifier graphql.String
+ State struct {
+ Name graphql.String
+ }
+ URL graphql.String
+}
+
+type findExistingIssuesSearch struct {
+ Issues struct {
+ Nodes []linearIssue
+ }
+}
+
+// Custom transport to add the API key to the header
+type addHeaderTransport struct {
+ T http.RoundTripper
+ Key string
+}
+
+func (adt *addHeaderTransport) RoundTrip(req *http.Request) (*http.Response, error) {
+ req.Header.Add("Authorization", adt.Key)
+ return adt.T.RoundTrip(req)
+}
+
+const (
+ linearPriorityNone = float64(0)
+ linearPriorityCritical = float64(1)
+ linearPriorityHigh = float64(2)
+ linearPriorityMedium = float64(3)
+ linearPriorityLow = float64(4)
+)
+
+// errors represents the "errors" array in a response from a GraphQL server.
+// If returned via error interface, the slice is expected to contain at least 1 element.
+//
+// Specification: https://spec.graphql.org/October2021/#sec-Errors.
+type errorsGraphql []struct {
+ Message string
+ Locations []struct {
+ Line int
+ Column int
+ }
+}
+
+// Error implements error interface.
+func (e errorsGraphql) Error() string {
+ return e[0].Message
+}
+
+// do executes a single GraphQL operation.
+func (i *Integration) doGraphqlRequest(ctx context.Context, query string, v any, variables map[string]any, operationName string) error {
+ in := struct {
+ Query string `json:"query"`
+ Variables map[string]any `json:"variables,omitempty"`
+ OperationName string `json:"operationName,omitempty"`
+ }{
+ Query: query,
+ Variables: variables,
+ OperationName: operationName,
+ }
+
+ var buf bytes.Buffer
+ err := json.NewEncoder(&buf).Encode(in)
+ if err != nil {
+ return err
+ }
+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, i.url, &buf)
+ if err != nil {
+ return err
+ }
+ req.Header.Set("Content-Type", "application/json")
+ resp, err := i.httpclient.Do(req)
+ if err != nil {
+ return err
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ body, _ := io.ReadAll(resp.Body)
+ return fmt.Errorf("non-200 OK status code: %v body: %q", resp.Status, body)
+ }
+ var out struct {
+ Data *json.RawMessage
+ Errors errorsGraphql
+ //Extensions any // Unused.
+ }
+
+ err = json.NewDecoder(resp.Body).Decode(&out)
+ if err != nil {
+ return err
+ }
+ if out.Data != nil {
+ err := jsonutil.UnmarshalGraphQL(*out.Data, v)
+ if err != nil {
+ return err
+ }
+ }
+ if len(out.Errors) > 0 {
+ return out.Errors
+ }
+ return nil
+}
diff --git a/pkg/scan/charts/echarts.go b/pkg/scan/charts/echarts.go
index f46dd365e4..298ce27a70 100644
--- a/pkg/scan/charts/echarts.go
+++ b/pkg/scan/charts/echarts.go
@@ -30,6 +30,7 @@ func (s *ScanEventsCharts) GenerateHTML(filePath string) error {
if err != nil {
return err
}
+ defer output.Close()
return page.Render(output)
}
diff --git a/pkg/scan/events/scan_noop.go b/pkg/scan/events/scan_noop.go
index a284657f52..055baed4ee 100644
--- a/pkg/scan/events/scan_noop.go
+++ b/pkg/scan/events/scan_noop.go
@@ -9,3 +9,6 @@ func AddScanEvent(event ScanEvent) {
func InitWithConfig(config *ScanConfig, statsDirectory string) {
}
+
+func Close() {
+}
diff --git a/pkg/scan/events/stats_build.go b/pkg/scan/events/stats_build.go
index 0f01724411..77f471df57 100644
--- a/pkg/scan/events/stats_build.go
+++ b/pkg/scan/events/stats_build.go
@@ -23,6 +23,7 @@ type ScanStatsWorker struct {
config *ScanConfig
m *sync.Mutex
directory string
+ file *os.File
enc *json.Encoder
}
@@ -56,6 +57,7 @@ func (s *ScanStatsWorker) initEventsFile() error {
if err != nil {
return err
}
+ s.file = f
s.enc = json.NewEncoder(f)
return nil
}
@@ -78,3 +80,22 @@ func AddScanEvent(event ScanEvent) {
}
defaultWorker.AddScanEvent(event)
}
+
+// Close closes the file associated with the worker
+func (s *ScanStatsWorker) Close() {
+ s.m.Lock()
+ defer s.m.Unlock()
+
+ if s.file != nil {
+ _ = s.file.Close()
+ s.file = nil
+ }
+}
+
+// Close closes the file associated with the worker
+func Close() {
+ if defaultWorker == nil {
+ return
+ }
+ defaultWorker.Close()
+}
diff --git a/pkg/scan/scan_context.go b/pkg/scan/scan_context.go
index 621d929512..6af1a687f5 100644
--- a/pkg/scan/scan_context.go
+++ b/pkg/scan/scan_context.go
@@ -6,8 +6,14 @@ import (
"strings"
"sync"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/contextargs"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/utils/errkit"
+>>>>>>> projectdiscovery-main
)
type ScanContextOption func(*ScanContext)
@@ -30,7 +36,7 @@ type ScanContext struct {
OnWarning func(string)
// unexported state fields
- errors []error
+ error error
warnings []string
events []*output.InternalWrappedEvent
results []*output.ResultEvent
@@ -52,6 +58,10 @@ func (s *ScanContext) Context() context.Context {
return s.ctx
}
+func (s *ScanContext) GenerateErrorMessage() error {
+ return s.error
+}
+
// GenerateResult returns final results slice from all events
func (s *ScanContext) GenerateResult() []*output.ResultEvent {
s.m.Lock()
@@ -77,6 +87,9 @@ func (s *ScanContext) LogEvent(e *output.InternalWrappedEvent) {
s.events = append(s.events, e)
}
+ e.RLock()
+ defer e.RUnlock()
+
s.results = append(s.results, e.Results...)
}
@@ -87,13 +100,16 @@ func (s *ScanContext) LogError(err error) {
if err == nil {
return
}
-
if s.OnError != nil {
s.OnError(err)
}
- s.errors = append(s.errors, err)
+ if s.error == nil {
+ s.error = err
+ } else {
+ s.error = errkit.Append(s.error, err)
+ }
- errorMessage := joinErrors(s.errors)
+ errorMessage := s.GenerateErrorMessage().Error()
for _, result := range s.results {
result.Error = errorMessage
@@ -122,14 +138,3 @@ func (s *ScanContext) LogWarning(format string, args ...any) {
}
}
}
-
-// joinErrors joins multiple errors and returns a single error string
-func joinErrors(errors []error) string {
- var errorMessages []string
- for _, e := range errors {
- if e != nil {
- errorMessages = append(errorMessages, e.Error())
- }
- }
- return strings.Join(errorMessages, "; ")
-}
diff --git a/pkg/templates/cluster.go b/pkg/templates/cluster.go
index be2dd94c7e..cfe615805f 100644
--- a/pkg/templates/cluster.go
+++ b/pkg/templates/cluster.go
@@ -13,8 +13,27 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/scan"
"github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/templates/cluster.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/contextargs"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/helpers/writer"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/writer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/templates/cluster.go
+>>>>>>> projectdiscovery-main
cryptoutil "github.com/projectdiscovery/utils/crypto"
- mapsutil "github.com/projectdiscovery/utils/maps"
)
// Cluster clusters a list of templates into a lesser number if possible based
@@ -42,91 +61,68 @@ import (
// Finally, the engine creates a single executer with a clusteredexecuter for all templates
// in a cluster.
func Cluster(list []*Template) [][]*Template {
+ http := make(map[uint64][]*Template)
+ dns := make(map[uint64][]*Template)
+ ssl := make(map[uint64][]*Template)
+
final := [][]*Template{}
- skip := mapsutil.NewSyncLockMap[string, struct{}]()
+ // Split up templates that might be clusterable
for _, template := range list {
- key := template.Path
-
- if skip.Has(key) {
- continue
- }
-
- // We only cluster http, dns and ssl requests as of now.
- // Take care of requests that can't be clustered first.
- if len(template.RequestsHTTP) == 0 && len(template.RequestsDNS) == 0 && len(template.RequestsSSL) == 0 {
- _ = skip.Set(key, struct{}{})
- final = append(final, []*Template{template})
- continue
- }
-
// it is not possible to cluster flow and multiprotocol due to dependent execution
if template.Flow != "" || template.Options.IsMultiProtocol {
- _ = skip.Set(key, struct{}{})
final = append(final, []*Template{template})
continue
}
- _ = skip.Set(key, struct{}{})
-
- var templateType types.ProtocolType
switch {
case len(template.RequestsDNS) == 1:
- templateType = types.DNSProtocol
- case len(template.RequestsHTTP) == 1:
- templateType = types.HTTPProtocol
- case len(template.RequestsSSL) == 1:
- templateType = types.SSLProtocol
- }
-
- // Find any/all similar matching request that is identical to
- // this one and cluster them together for http protocol only.
- cluster := []*Template{}
- for _, other := range list {
- otherKey := other.Path
-
- if skip.Has(otherKey) {
- continue
- }
-
- // it is not possible to cluster flow and multiprotocol due to dependent execution
- if other.Flow != "" || other.Options.IsMultiProtocol {
- _ = skip.Set(otherKey, struct{}{})
- final = append(final, []*Template{other})
- continue
+ if template.RequestsDNS[0].IsClusterable() {
+ hash := template.RequestsDNS[0].TmplClusterKey()
+ if dns[hash] == nil {
+ dns[hash] = []*Template{}
+ }
+ dns[hash] = append(dns[hash], template)
+ } else {
+ final = append(final, []*Template{template})
}
- switch templateType {
- case types.DNSProtocol:
- if len(other.RequestsDNS) != 1 {
- continue
- } else if template.RequestsDNS[0].CanCluster(other.RequestsDNS[0]) {
- _ = skip.Set(otherKey, struct{}{})
- cluster = append(cluster, other)
- }
- case types.HTTPProtocol:
- if len(other.RequestsHTTP) != 1 {
- continue
- } else if template.RequestsHTTP[0].CanCluster(other.RequestsHTTP[0]) {
- _ = skip.Set(otherKey, struct{}{})
- cluster = append(cluster, other)
+ case len(template.RequestsHTTP) == 1:
+ if template.RequestsHTTP[0].IsClusterable() {
+ hash := template.RequestsHTTP[0].TmplClusterKey()
+ if http[hash] == nil {
+ http[hash] = []*Template{}
}
- case types.SSLProtocol:
- if len(other.RequestsSSL) != 1 {
- continue
- } else if template.RequestsSSL[0].CanCluster(other.RequestsSSL[0]) {
- _ = skip.Set(otherKey, struct{}{})
- cluster = append(cluster, other)
+ http[hash] = append(http[hash], template)
+ } else {
+ final = append(final, []*Template{template})
+ }
+ case len(template.RequestsSSL) == 1:
+ if template.RequestsSSL[0].IsClusterable() {
+ hash := template.RequestsSSL[0].TmplClusterKey()
+ if ssl[hash] == nil {
+ ssl[hash] = []*Template{}
}
+ ssl[hash] = append(ssl[hash], template)
+ } else {
+ final = append(final, []*Template{template})
}
- }
- if len(cluster) > 0 {
- cluster = append(cluster, template)
- final = append(final, cluster)
- } else {
+ default:
final = append(final, []*Template{template})
}
}
+
+ // add all clusterd templates
+ for _, templates := range http {
+ final = append(final, templates)
+ }
+ for _, templates := range dns {
+ final = append(final, templates)
+ }
+ for _, templates := range ssl {
+ final = append(final, templates)
+ }
+
return final
}
@@ -275,28 +271,30 @@ func (e *ClusterExecuter) Execute(ctx *scan.ScanContext) (bool, error) {
event.InternalEvent = make(map[string]interface{})
}
for _, operator := range e.operators {
- result, matched := operator.operator.Execute(event.InternalEvent, e.requests.Match, e.requests.Extract, e.options.Options.Debug || e.options.Options.DebugResponse)
- event.InternalEvent["template-id"] = operator.templateID
- event.InternalEvent["template-path"] = operator.templatePath
- event.InternalEvent["template-info"] = operator.templateInfo
+ clonedEvent := event.CloneShallow()
+
+ result, matched := operator.operator.Execute(clonedEvent.InternalEvent, e.requests.Match, e.requests.Extract, e.options.Options.Debug || e.options.Options.DebugResponse)
+ clonedEvent.InternalEvent["template-id"] = operator.templateID
+ clonedEvent.InternalEvent["template-path"] = operator.templatePath
+ clonedEvent.InternalEvent["template-info"] = operator.templateInfo
if result == nil && !matched && e.options.Options.MatcherStatus {
- if err := e.options.Output.WriteFailure(event); err != nil {
+ if err := e.options.Output.WriteFailure(clonedEvent); err != nil {
gologger.Warning().Msgf("Could not write failure event to output: %s\n", err)
}
continue
}
if matched && result != nil {
- event.OperatorsResult = result
- event.Results = e.requests.MakeResultEvent(event)
+ clonedEvent.OperatorsResult = result
+ clonedEvent.Results = e.requests.MakeResultEvent(clonedEvent)
results = true
- _ = writer.WriteResult(event, e.options.Output, e.options.Progress, e.options.IssuesClient)
+ _ = writer.WriteResult(clonedEvent, e.options.Output, e.options.Progress, e.options.IssuesClient)
}
}
})
if err != nil && e.options.HostErrorsCache != nil {
- e.options.HostErrorsCache.MarkFailed(ctx.Input.MetaInput.Input, err)
+ e.options.HostErrorsCache.MarkFailed(e.options.ProtocolType.String(), ctx.Input, err)
}
return results, err
}
@@ -314,14 +312,16 @@ func (e *ClusterExecuter) ExecuteWithResults(ctx *scan.ScanContext) ([]*output.R
}
err := e.requests.ExecuteWithResults(inputItem, dynamicValues, nil, func(event *output.InternalWrappedEvent) {
for _, operator := range e.operators {
- result, matched := operator.operator.Execute(event.InternalEvent, e.requests.Match, e.requests.Extract, e.options.Options.Debug || e.options.Options.DebugResponse)
+ clonedEvent := event.CloneShallow()
+
+ result, matched := operator.operator.Execute(clonedEvent.InternalEvent, e.requests.Match, e.requests.Extract, e.options.Options.Debug || e.options.Options.DebugResponse)
if matched && result != nil {
- event.OperatorsResult = result
- event.InternalEvent["template-id"] = operator.templateID
- event.InternalEvent["template-path"] = operator.templatePath
- event.InternalEvent["template-info"] = operator.templateInfo
- event.Results = e.requests.MakeResultEvent(event)
- scanCtx.LogEvent(event)
+ clonedEvent.OperatorsResult = result
+ clonedEvent.InternalEvent["template-id"] = operator.templateID
+ clonedEvent.InternalEvent["template-path"] = operator.templatePath
+ clonedEvent.InternalEvent["template-info"] = operator.templateInfo
+ clonedEvent.Results = e.requests.MakeResultEvent(clonedEvent)
+ scanCtx.LogEvent(clonedEvent)
}
}
})
@@ -330,7 +330,7 @@ func (e *ClusterExecuter) ExecuteWithResults(ctx *scan.ScanContext) ([]*output.R
}
if err != nil && e.options.HostErrorsCache != nil {
- e.options.HostErrorsCache.MarkFailed(ctx.Input.MetaInput.Input, err)
+ e.options.HostErrorsCache.MarkFailed(e.options.ProtocolType.String(), ctx.Input, err)
}
return scanCtx.GenerateResult(), err
}
diff --git a/pkg/templates/compile.go b/pkg/templates/compile.go
index a5c02415cd..0937a299b9 100644
--- a/pkg/templates/compile.go
+++ b/pkg/templates/compile.go
@@ -24,6 +24,20 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/tmplexec"
"github.com/Explorer1092/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/js/compiler"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/globalmatchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/offlinehttp"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/signer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/tmplexec"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
stringsutil "github.com/projectdiscovery/utils/strings"
)
@@ -36,8 +50,7 @@ var (
)
const (
- Unsigned = "unsigned"
- PDVerifier = "projectdiscovery/nuclei-templates"
+ Unsigned = "unsigned"
)
func init() {
@@ -82,6 +95,18 @@ func Parse(filePath string, preprocessor Preprocessor, options protocols.Executo
if err != nil {
return nil, err
}
+ if template.isGlobalMatchersEnabled() {
+ item := &globalmatchers.Item{
+ TemplateID: template.ID,
+ TemplatePath: filePath,
+ TemplateInfo: template.Info,
+ }
+ for _, request := range template.RequestsHTTP {
+ item.Operators = append(item.Operators, request.CompiledOperators)
+ }
+ options.GlobalMatchers.AddOperator(item)
+ return nil, nil
+ }
// Compile the workflow request
if len(template.Workflows) > 0 {
compiled := &template.Workflow
@@ -97,6 +122,25 @@ func Parse(filePath string, preprocessor Preprocessor, options protocols.Executo
return template, nil
}
+// isGlobalMatchersEnabled checks if any of requests in the template
+// have global matchers enabled. It iterates through all requests and
+// returns true if at least one request has global matchers enabled;
+// otherwise, it returns false.
+//
+// Note: This method only checks the `RequestsHTTP`
+// field of the template, which is specific to http-protocol-based
+// templates.
+//
+// TODO: support all protocols.
+func (template *Template) isGlobalMatchersEnabled() bool {
+ for _, request := range template.RequestsHTTP {
+ if request.GlobalMatchers {
+ return true
+ }
+ }
+ return false
+}
+
// parseSelfContainedRequests parses the self contained template requests.
func (template *Template) parseSelfContainedRequests() {
if template.Signature.Value.String() != "" {
@@ -342,6 +386,11 @@ func parseTemplate(data []byte, options protocols.ExecutorOptions) (*Template, e
return nil, errors.New("no template author field provided")
}
+ numberOfWorkflows := len(template.Workflows)
+ if numberOfWorkflows > 0 && numberOfWorkflows != template.Requests() {
+ return nil, errors.New("workflows cannot have other protocols")
+ }
+
// use default unknown severity
if len(template.Workflows) == 0 {
if template.Info.SeverityHolder.Severity == severity.Undefined {
@@ -415,7 +464,7 @@ func parseTemplate(data []byte, options protocols.ExecutorOptions) (*Template, e
break
}
}
-
+ options.TemplateVerifier = template.TemplateVerifier
if !(template.Verified && verifier.Identifier() == "projectdiscovery/nuclei-templates") {
template.Options.RawTemplate = data
}
diff --git a/pkg/templates/compile_test.go b/pkg/templates/compile_test.go
index f29489554d..1d77fcfa43 100644
--- a/pkg/templates/compile_test.go
+++ b/pkg/templates/compile_test.go
@@ -27,6 +27,44 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
"github.com/Explorer1092/nuclei/v3/pkg/workflows"
"github.com/julienschmidt/httprouter"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/templates/compile_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/disk"
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/parsers"
+ "github.com/Explorer1092/nuclei/v2/pkg/progress"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/generators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/variables"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates"
+ "github.com/Explorer1092/nuclei/v2/pkg/testutils"
+ "github.com/Explorer1092/nuclei/v2/pkg/workflows"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/loader/workflow"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/progress"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/variables"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/workflows"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/templates/compile_test.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/ratelimit"
"github.com/stretchr/testify/require"
)
@@ -39,15 +77,15 @@ func setup() {
progressImpl, _ := progress.NewStatsTicker(0, false, false, false, 0)
executerOpts = protocols.ExecutorOptions{
- Output: testutils.NewMockOutputWriter(options.OmitTemplate),
- Options: options,
- Progress: progressImpl,
- ProjectFile: nil,
- IssuesClient: nil,
- Browser: nil,
- Catalog: disk.NewCatalog(config.DefaultConfig.TemplatesDirectory),
- RateLimiter: ratelimit.New(context.Background(), uint(options.RateLimit), time.Second),
- Parser: templates.NewParser(),
+ Output: testutils.NewMockOutputWriter(options.OmitTemplate),
+ Options: options,
+ Progress: progressImpl,
+ ProjectFile: nil,
+ IssuesClient: nil,
+ Browser: nil,
+ Catalog: disk.NewCatalog(config.DefaultConfig.TemplatesDirectory),
+ RateLimiter: ratelimit.New(context.Background(), uint(options.RateLimit), time.Second),
+ Parser: templates.NewParser(),
}
workflowLoader, err := workflow.NewLoader(&executerOpts)
if err != nil {
@@ -197,3 +235,12 @@ func Test_WrongTemplate(t *testing.T) {
require.Nil(t, got, "could not parse template")
require.ErrorContains(t, err, "no requests defined ")
}
+
+func TestWrongWorkflow(t *testing.T) {
+ setup()
+
+ filePath := "tests/workflow-invalid.yaml"
+ got, err := templates.Parse(filePath, nil, executerOpts)
+ require.Nil(t, got, "could not parse template")
+ require.ErrorContains(t, err, "workflows cannot have other protocols")
+}
diff --git a/pkg/templates/parser_stats.go b/pkg/templates/parser_stats.go
index 290601032d..119746adcb 100644
--- a/pkg/templates/parser_stats.go
+++ b/pkg/templates/parser_stats.go
@@ -10,5 +10,7 @@ const (
ExcludedCodeTmplStats = "code-flag-missing-warnings"
ExludedDastTmplStats = "fuzz-flag-missing-warnings"
SkippedUnsignedStats = "skipped-unsigned-stats" // tracks loading of unsigned templates
+ ExcludedSelfContainedStats = "excluded-self-contained-stats"
+ ExcludedFileStats = "excluded-file-stats"
SkippedRequestSignatureStats = "skipped-request-signature-stats"
)
diff --git a/pkg/templates/parser_test.go b/pkg/templates/parser_test.go
index 9eb0a7ffc1..51db2a2e6f 100644
--- a/pkg/templates/parser_test.go
+++ b/pkg/templates/parser_test.go
@@ -5,10 +5,34 @@ import (
"fmt"
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/disk"
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
+=======
+<<<<<<< HEAD:v2/pkg/parsers/parser_test.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/disk"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/loader/filter"
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/catalog/disk"
+ "github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader/filter"
+ "github.com/projectdiscovery/nuclei/v2/pkg/model"
+ "github.com/projectdiscovery/nuclei/v2/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v2/pkg/templates"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/templates/parser_test.go
+>>>>>>> projectdiscovery-main
"github.com/stretchr/testify/require"
)
@@ -52,7 +76,15 @@ func TestLoadTemplate(t *testing.T) {
SeverityHolder: severity.Holder{Severity: severity.Medium},
},
},
+<<<<<<< HEAD
expectedErr: errors.New("mandatory 'name' field is missing\ninvalid field format for 'id' (allowed format is ^([a-zA-Z0-9\p{Han}\!\(\)\.]+[-_])*[a-zA-Z0-9\p{Han}\!\(\)\.]+$ "),
+=======
+<<<<<<< HEAD:v2/pkg/parsers/parser_test.go
+ expectedErr: errors.New(`mandatory 'name' field is missing, invalid field format for 'id' (allowed format is ^([a-zA-Z0-9\p{Han}\!\(\)\.]+[-_])*[a-zA-Z0-9\p{Han}\!\(\)\.]+$)`),
+=======
+ expectedErr: errors.New("mandatory 'name' field is missing\ninvalid field format for 'id' (allowed format is ^([a-zA-Z0-9]+[-_])*[a-zA-Z0-9]+$)"),
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/templates/parser_test.go
+>>>>>>> projectdiscovery-main
},
{
name: "emptySeverity",
diff --git a/pkg/templates/preprocessors.go b/pkg/templates/preprocessors.go
index 054a0bc41e..7cf6f8e21a 100644
--- a/pkg/templates/preprocessors.go
+++ b/pkg/templates/preprocessors.go
@@ -5,6 +5,7 @@ import (
"regexp"
"strings"
+ stringsutil "github.com/projectdiscovery/utils/strings"
"github.com/segmentio/ksuid"
)
@@ -48,7 +49,7 @@ func (r *randStrPreprocessor) ProcessNReturnData(data []byte) ([]byte, map[strin
continue
}
value := expression[1]
- if strings.Contains(value, "(") || strings.Contains(value, ")") {
+ if stringsutil.ContainsAny(value, "(", ")") {
continue
}
diff --git a/pkg/templates/signer/tmpl_signer.go b/pkg/templates/signer/tmpl_signer.go
index 4c40286973..c3ee740b51 100644
--- a/pkg/templates/signer/tmpl_signer.go
+++ b/pkg/templates/signer/tmpl_signer.go
@@ -11,7 +11,6 @@ import (
"errors"
"fmt"
"os"
- "regexp"
"strings"
"sync"
@@ -21,18 +20,21 @@ import (
)
var (
- ReDigest = regexp.MustCompile(`(?m)^#\sdigest:\s.+$`)
ErrUnknownAlgorithm = errors.New("unknown algorithm")
SignaturePattern = "# digest: "
SignatureFmt = SignaturePattern + "%x" + ":%v" // `#digest: :`
)
-func RemoveSignatureFromData(data []byte) []byte {
- return bytes.Trim(ReDigest.ReplaceAll(data, []byte("")), "\n")
-}
-
-func GetSignatureFromData(data []byte) []byte {
- return ReDigest.Find(data)
+// ExtractSignatureAndContent extracts the signature (if present) and returns the content without the signature
+func ExtractSignatureAndContent(data []byte) (signature, content []byte) {
+ dataStr := string(data)
+ if idx := strings.LastIndex(dataStr, SignaturePattern); idx != -1 {
+ signature = []byte(strings.TrimSpace(dataStr[idx:]))
+ content = []byte(strings.TrimSpace(dataStr[:idx]))
+ } else {
+ content = data
+ }
+ return
}
// SignableTemplate is a template that can be signed
@@ -69,26 +71,29 @@ func (t *TemplateSigner) GetUserFragment() string {
// Sign signs the given template with the template signer and returns the signature
func (t *TemplateSigner) Sign(data []byte, tmpl SignableTemplate) (string, error) {
+ existingSignature, content := ExtractSignatureAndContent(data)
+
// while re-signing template check if it has a code protocol
// if it does then verify that it is signed by current signer
// if not then return error
if tmpl.HasCodeProtocol() {
- sig := GetSignatureFromData(data)
- arr := strings.SplitN(string(sig), ":", 3)
- if len(arr) == 2 {
- // signature has no fragment
- return "", errorutil.NewWithTag("signer", "re-signing code templates are not allowed for security reasons.")
- }
- if len(arr) == 3 {
- // signature has fragment verify if it is equal to current fragment
- fragment := t.GetUserFragment()
- if fragment != arr[2] {
+ if len(existingSignature) > 0 {
+ arr := strings.SplitN(string(existingSignature), ":", 3)
+ if len(arr) == 2 {
+ // signature has no fragment
return "", errorutil.NewWithTag("signer", "re-signing code templates are not allowed for security reasons.")
}
+ if len(arr) == 3 {
+ // signature has fragment verify if it is equal to current fragment
+ fragment := t.GetUserFragment()
+ if fragment != arr[2] {
+ return "", errorutil.NewWithTag("signer", "re-signing code templates are not allowed for security reasons.")
+ }
+ }
}
}
- buff := bytes.NewBuffer(RemoveSignatureFromData(data))
+ buff := bytes.NewBuffer(content)
// if file has any imports process them
for _, file := range tmpl.GetFileImports() {
bin, err := os.ReadFile(file)
@@ -123,12 +128,16 @@ func (t *TemplateSigner) sign(data []byte) (string, error) {
// Verify verifies the given template with the template signer
func (t *TemplateSigner) Verify(data []byte, tmpl SignableTemplate) (bool, error) {
- digestData := ReDigest.Find(data)
- if len(digestData) == 0 {
- return false, errors.New("digest not found")
+ signature, content := ExtractSignatureAndContent(data)
+ if len(signature) == 0 {
+ return false, errors.New("no signature found")
+ }
+
+ if !bytes.HasPrefix(signature, []byte(SignaturePattern)) {
+ return false, errors.New("signature must be at the end of the template")
}
- digestData = bytes.TrimSpace(bytes.TrimPrefix(digestData, []byte(SignaturePattern)))
+ digestData := bytes.TrimSpace(bytes.TrimPrefix(signature, []byte(SignaturePattern)))
// remove fragment from digest as it is used for re-signing purposes only
digestString := strings.TrimSuffix(string(digestData), ":"+t.GetUserFragment())
digest, err := hex.DecodeString(digestString)
@@ -136,7 +145,7 @@ func (t *TemplateSigner) Verify(data []byte, tmpl SignableTemplate) (bool, error
return false, err
}
- buff := bytes.NewBuffer(RemoveSignatureFromData(data))
+ buff := bytes.NewBuffer(content)
// if file has any imports process them
for _, file := range tmpl.GetFileImports() {
bin, err := os.ReadFile(file)
diff --git a/pkg/templates/signer/tmpl_signer_test.go b/pkg/templates/signer/tmpl_signer_test.go
new file mode 100644
index 0000000000..832dcb0e05
--- /dev/null
+++ b/pkg/templates/signer/tmpl_signer_test.go
@@ -0,0 +1,126 @@
+package signer
+
+import (
+ "bytes"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+const (
+ testCertFile = "../../../integration_tests/protocols/keys/ci.crt"
+ testKeyFile = "../../../integration_tests/protocols/keys/ci-private-key.pem"
+)
+
+type mockSignableTemplate struct {
+ imports []string
+ hasCode bool
+}
+
+func (m *mockSignableTemplate) GetFileImports() []string {
+ return m.imports
+}
+
+func (m *mockSignableTemplate) HasCodeProtocol() bool {
+ return m.hasCode
+}
+
+var signer, _ = NewTemplateSignerFromFiles(testCertFile, testKeyFile)
+
+func TestTemplateSignerSignAndVerify(t *testing.T) {
+ tempDir := t.TempDir()
+
+ tests := []struct {
+ name string
+ data []byte
+ tmpl SignableTemplate
+ wantSignErr bool
+ wantVerifyErr bool
+ wantVerified bool
+ modifyAfterSign func([]byte) []byte
+ }{
+ {
+ name: "Simple template",
+ data: []byte("id: test-template\ninfo:\n name: Test Template"),
+ tmpl: &mockSignableTemplate{},
+ wantVerified: true,
+ },
+ {
+ name: "Template with imports",
+ data: []byte("id: test-template\ninfo:\n name: Test Template"),
+ tmpl: &mockSignableTemplate{imports: []string{
+ filepath.Join(tempDir, "import1.yaml"),
+ filepath.Join(tempDir, "import2.yaml"),
+ }},
+ wantVerified: true,
+ },
+ {
+ name: "Template with code protocol",
+ data: []byte("id: test-template\ninfo:\n name: Test Template\n\ncode:\n - engine: bash\n source: echo 'Hello, World!'"),
+ tmpl: &mockSignableTemplate{hasCode: true},
+ wantSignErr: false,
+ wantVerified: true,
+ },
+ {
+ name: "Tampered template",
+ data: []byte("id: test-template\ninfo:\n name: Test Template"),
+ tmpl: &mockSignableTemplate{},
+ modifyAfterSign: func(data []byte) []byte {
+ signatureIndex := bytes.LastIndex(data, []byte(SignaturePattern))
+ if signatureIndex == -1 {
+ return data
+ }
+ return append(data[:signatureIndex], append([]byte("# Tampered content\n"), data[signatureIndex:]...)...)
+ },
+ wantVerified: false,
+ },
+ {
+ name: "Invalid signature",
+ data: []byte("id: test-template\ninfo:\n name: Test Template"),
+ tmpl: &mockSignableTemplate{},
+ modifyAfterSign: func(data []byte) []byte {
+ return append(bytes.TrimSuffix(data, []byte("\n")), []byte("\n# digest: invalid_signature:fragment")...)
+ },
+ wantVerifyErr: true,
+ wantVerified: false,
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ // Create import files if needed
+ for _, imp := range tt.tmpl.GetFileImports() {
+ err := os.WriteFile(imp, []byte("imported content"), 0644)
+ require.NoError(t, err, "Failed to create import file")
+ }
+
+ // Sign the template
+ signature, err := signer.Sign(tt.data, tt.tmpl)
+ if tt.wantSignErr {
+ assert.Error(t, err, "Expected an error during signing")
+ return
+ }
+ require.NoError(t, err, "Failed to sign template")
+
+ // Append signature to the template data
+ signedData := append(tt.data, []byte("\n"+signature)...)
+
+ // Apply any modifications after signing if specified
+ if tt.modifyAfterSign != nil {
+ signedData = tt.modifyAfterSign(signedData)
+ }
+
+ // Verify the signature
+ verified, err := signer.Verify(signedData, tt.tmpl)
+ if tt.wantVerifyErr {
+ assert.Error(t, err, "Expected an error during verification")
+ } else {
+ assert.NoError(t, err, "Unexpected error during verification")
+ }
+ assert.Equal(t, tt.wantVerified, verified, "Unexpected verification result")
+ })
+ }
+}
diff --git a/pkg/templates/stats.go b/pkg/templates/stats.go
index 3619909648..b1829404b5 100644
--- a/pkg/templates/stats.go
+++ b/pkg/templates/stats.go
@@ -9,6 +9,8 @@ func init() {
stats.NewEntry(SkippedCodeTmplTamperedStats, "Found %d unsigned or tampered code template (carefully examine before using it & use -sign flag to sign them)")
stats.NewEntry(ExcludedHeadlessTmplStats, "Excluded %d headless template[s] (disabled as default), use -headless option to run headless templates.")
stats.NewEntry(ExcludedCodeTmplStats, "Excluded %d code template[s] (disabled as default), use -code option to run code templates.")
+ stats.NewEntry(ExcludedSelfContainedStats, "Excluded %d self-contained template[s] (disabled as default), use -esc option to run self-contained templates.")
+ stats.NewEntry(ExcludedFileStats, "Excluded %d file template[s] (disabled as default), use -file option to run file templates.")
stats.NewEntry(TemplatesExcludedStats, "Excluded %d template[s] with known weak matchers / tags excluded from default run using .nuclei-ignore")
stats.NewEntry(ExludedDastTmplStats, "Excluded %d dast template[s] (disabled as default), use -dast option to run dast templates.")
stats.NewEntry(SkippedUnsignedStats, "Skipping %d unsigned template[s]")
diff --git a/pkg/templates/tag_filter.go b/pkg/templates/tag_filter.go
index c23235c57f..9d231f1899 100644
--- a/pkg/templates/tag_filter.go
+++ b/pkg/templates/tag_filter.go
@@ -15,6 +15,23 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/Knetic/govaluate"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/catalog/loader/filter/tag_filter.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/common/dsl"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/templates/tag_filter.go
+>>>>>>> projectdiscovery-main
sliceutil "github.com/projectdiscovery/utils/slice"
)
diff --git a/pkg/templates/tag_filter_test.go b/pkg/templates/tag_filter_test.go
index f593794c98..03e3287cc6 100644
--- a/pkg/templates/tag_filter_test.go
+++ b/pkg/templates/tag_filter_test.go
@@ -3,12 +3,40 @@ package templates
import (
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/dns"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/http"
"github.com/Explorer1092/nuclei/v3/pkg/templates/types"
+=======
+<<<<<<< HEAD:v2/pkg/catalog/loader/filter/tag_filter_test.go
+<<<<<<< HEAD
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/dns"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+=======
+ "github.com/projectdiscovery/nuclei/v2/pkg/model"
+ "github.com/projectdiscovery/nuclei/v2/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/dns"
+ "github.com/projectdiscovery/nuclei/v2/pkg/protocols/http"
+ "github.com/projectdiscovery/nuclei/v2/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
+>>>>>>> bb98eced070f4ae137b8cd2a7f887611bc1b9c93
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/dns"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/templates/tag_filter_test.go
+>>>>>>> projectdiscovery-main
"github.com/stretchr/testify/require"
)
diff --git a/pkg/templates/template_sign.go b/pkg/templates/template_sign.go
index 3b76001d8b..0a7b948dbe 100644
--- a/pkg/templates/template_sign.go
+++ b/pkg/templates/template_sign.go
@@ -75,11 +75,12 @@ func SignTemplate(templateSigner *signer.TemplateSigner, templatePath string) er
return ErrNotATemplate
}
if !template.Verified {
+ _, content := signer.ExtractSignatureAndContent(bin)
signatureData, err := templateSigner.Sign(bin, template)
if err != nil {
return err
}
- buff := bytes.NewBuffer(signer.RemoveSignatureFromData(bin))
+ buff := bytes.NewBuffer(content)
buff.WriteString("\n" + signatureData)
return os.WriteFile(templatePath, buff.Bytes(), 0644)
}
diff --git a/pkg/templates/templates.go b/pkg/templates/templates.go
index 5b8f9afb75..93ea2facd6 100644
--- a/pkg/templates/templates.go
+++ b/pkg/templates/templates.go
@@ -9,6 +9,7 @@ import (
"strings"
validate "github.com/go-playground/validator/v10"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/code"
@@ -24,10 +25,27 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/whois"
"github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/Explorer1092/nuclei/v3/pkg/workflows"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/code"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/variables"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/dns"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/file"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/javascript"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/network"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/ssl"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/websocket"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/whois"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/workflows"
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
"go.uber.org/multierr"
- "gopkg.in/yaml.v2"
)
// Template is a YAML input file which defines all the requests and
@@ -45,7 +63,11 @@ type Template struct {
// examples:
// - name: ID Example
// value: "\"CVE-2021-19520\""
+<<<<<<< HEAD
ID string `yaml:"id" json:"id" jsonschema:"title=id of the template,description=The Unique ID for the template,required,example=cve-2021-19520,pattern=^([a-zA-Z0-9\p{Han}]+[-_])*[a-zA-Z0-9\p{Han}]+$"`
+=======
+ ID string `yaml:"id" json:"id" jsonschema:"title=id of the template,description=The Unique ID for the template,required,example=cve-2021-19520,pattern=^([a-zA-Z0-9\p{Han}\!\(\)\.]+[-_])*[a-zA-Z0-9\p{Han}\!\(\)\.]+$"`
+>>>>>>> projectdiscovery-main
// description: |
// Info contains metadata information about the template.
// examples:
@@ -176,8 +198,6 @@ func (template *Template) Type() types.ProtocolType {
return types.HeadlessProtocol
case len(template.RequestsNetwork) > 0:
return types.NetworkProtocol
- case len(template.Workflow.Workflows) > 0:
- return types.WorkflowProtocol
case len(template.RequestsSSL) > 0:
return types.SSLProtocol
case len(template.RequestsWebsocket) > 0:
@@ -188,6 +208,8 @@ func (template *Template) Type() types.ProtocolType {
return types.CodeProtocol
case len(template.RequestsJavascript) > 0:
return types.JavascriptProtocol
+ case len(template.Workflow.Workflows) > 0:
+ return types.WorkflowProtocol
default:
return types.InvalidProtocol
}
@@ -326,6 +348,17 @@ func (template *Template) UnmarshalYAML(unmarshal func(interface{}) error) error
}
*template = Template(*alias)
+ if !ReTemplateID.MatchString(template.ID) {
+ return errorutil.New("template id must match expression %v", ReTemplateID).WithTag("invalid template")
+ }
+ info := template.Info
+ if utils.IsBlank(info.Name) {
+ return errorutil.New("no template name field provided").WithTag("invalid template")
+ }
+ if info.Authors.IsEmpty() {
+ return errorutil.New("no template author field provided").WithTag("invalid template")
+ }
+
if len(template.RequestsHTTP) > 0 || len(template.RequestsNetwork) > 0 {
_ = deprecatedProtocolNameTemplates.Set(template.ID, true)
}
@@ -543,3 +576,8 @@ func (template *Template) UnmarshalJSON(data []byte) error {
}
return nil
}
+
+// HasFileProtocol returns true if the template has a file protocol section
+func (template *Template) HasFileProtocol() bool {
+ return len(template.RequestsFile) > 0
+}
diff --git a/pkg/templates/templates_doc.go b/pkg/templates/templates_doc.go
index 42fa25b947..c6b2186ef8 100644
--- a/pkg/templates/templates_doc.go
+++ b/pkg/templates/templates_doc.go
@@ -20,6 +20,7 @@ var (
HTTPMethodTypeHolderDoc encoder.Doc
FUZZRuleDoc encoder.Doc
SliceOrMapSliceDoc encoder.Doc
+ ANALYZERSAnalyzerTemplateDoc encoder.Doc
SignatureTypeHolderDoc encoder.Doc
MATCHERSMatcherDoc encoder.Doc
MatcherTypeHolderDoc encoder.Doc
@@ -459,7 +460,7 @@ func init() {
Value: "HTTP response headers in name:value format",
},
}
- HTTPRequestDoc.Fields = make([]encoder.Doc, 35)
+ HTTPRequestDoc.Fields = make([]encoder.Doc, 38)
HTTPRequestDoc.Fields[0].Name = "path"
HTTPRequestDoc.Fields[0].Type = "[]string"
HTTPRequestDoc.Fields[0].Note = ""
@@ -565,104 +566,119 @@ func init() {
HTTPRequestDoc.Fields[15].Note = ""
HTTPRequestDoc.Fields[15].Description = "Fuzzing describes schema to fuzz http requests"
HTTPRequestDoc.Fields[15].Comments[encoder.LineComment] = " Fuzzing describes schema to fuzz http requests"
- HTTPRequestDoc.Fields[16].Name = "self-contained"
- HTTPRequestDoc.Fields[16].Type = "bool"
+ HTTPRequestDoc.Fields[16].Name = "analyzer"
+ HTTPRequestDoc.Fields[16].Type = "analyzers.AnalyzerTemplate"
HTTPRequestDoc.Fields[16].Note = ""
- HTTPRequestDoc.Fields[16].Description = "SelfContained specifies if the request is self-contained."
- HTTPRequestDoc.Fields[16].Comments[encoder.LineComment] = "SelfContained specifies if the request is self-contained."
- HTTPRequestDoc.Fields[17].Name = "signature"
- HTTPRequestDoc.Fields[17].Type = "SignatureTypeHolder"
+ HTTPRequestDoc.Fields[16].Description = "Analyzer is an analyzer to use for matching the response."
+ HTTPRequestDoc.Fields[16].Comments[encoder.LineComment] = "Analyzer is an analyzer to use for matching the response."
+ HTTPRequestDoc.Fields[17].Name = "self-contained"
+ HTTPRequestDoc.Fields[17].Type = "bool"
HTTPRequestDoc.Fields[17].Note = ""
- HTTPRequestDoc.Fields[17].Description = "Signature is the request signature method"
- HTTPRequestDoc.Fields[17].Comments[encoder.LineComment] = "Signature is the request signature method"
- HTTPRequestDoc.Fields[17].Values = []string{
+ HTTPRequestDoc.Fields[17].Description = "SelfContained specifies if the request is self-contained."
+ HTTPRequestDoc.Fields[17].Comments[encoder.LineComment] = "SelfContained specifies if the request is self-contained."
+ HTTPRequestDoc.Fields[18].Name = "signature"
+ HTTPRequestDoc.Fields[18].Type = "SignatureTypeHolder"
+ HTTPRequestDoc.Fields[18].Note = ""
+ HTTPRequestDoc.Fields[18].Description = "Signature is the request signature method"
+ HTTPRequestDoc.Fields[18].Comments[encoder.LineComment] = "Signature is the request signature method"
+ HTTPRequestDoc.Fields[18].Values = []string{
"AWS",
}
- HTTPRequestDoc.Fields[18].Name = "cookie-reuse"
- HTTPRequestDoc.Fields[18].Type = "bool"
- HTTPRequestDoc.Fields[18].Note = ""
- HTTPRequestDoc.Fields[18].Description = "CookieReuse is an optional setting that enables cookie reuse for\nall requests defined in raw section."
- HTTPRequestDoc.Fields[18].Comments[encoder.LineComment] = "CookieReuse is an optional setting that enables cookie reuse for"
- HTTPRequestDoc.Fields[19].Name = "disable-cookie"
+ HTTPRequestDoc.Fields[19].Name = "skip-secret-file"
HTTPRequestDoc.Fields[19].Type = "bool"
HTTPRequestDoc.Fields[19].Note = ""
- HTTPRequestDoc.Fields[19].Description = "DisableCookie is an optional setting that disables cookie reuse"
- HTTPRequestDoc.Fields[19].Comments[encoder.LineComment] = "DisableCookie is an optional setting that disables cookie reuse"
- HTTPRequestDoc.Fields[20].Name = "read-all"
+ HTTPRequestDoc.Fields[19].Description = "SkipSecretFile skips the authentication or authorization configured in the secret file."
+ HTTPRequestDoc.Fields[19].Comments[encoder.LineComment] = "SkipSecretFile skips the authentication or authorization configured in the secret file."
+ HTTPRequestDoc.Fields[20].Name = "cookie-reuse"
HTTPRequestDoc.Fields[20].Type = "bool"
HTTPRequestDoc.Fields[20].Note = ""
- HTTPRequestDoc.Fields[20].Description = "Enables force reading of the entire raw unsafe request body ignoring\nany specified content length headers."
- HTTPRequestDoc.Fields[20].Comments[encoder.LineComment] = "Enables force reading of the entire raw unsafe request body ignoring"
- HTTPRequestDoc.Fields[21].Name = "redirects"
+ HTTPRequestDoc.Fields[20].Description = "CookieReuse is an optional setting that enables cookie reuse for\nall requests defined in raw section."
+ HTTPRequestDoc.Fields[20].Comments[encoder.LineComment] = "CookieReuse is an optional setting that enables cookie reuse for"
+ HTTPRequestDoc.Fields[21].Name = "disable-cookie"
HTTPRequestDoc.Fields[21].Type = "bool"
HTTPRequestDoc.Fields[21].Note = ""
- HTTPRequestDoc.Fields[21].Description = "Redirects specifies whether redirects should be followed by the HTTP Client.\n\nThis can be used in conjunction with `max-redirects` to control the HTTP request redirects."
- HTTPRequestDoc.Fields[21].Comments[encoder.LineComment] = "Redirects specifies whether redirects should be followed by the HTTP Client."
- HTTPRequestDoc.Fields[22].Name = "host-redirects"
+ HTTPRequestDoc.Fields[21].Description = "DisableCookie is an optional setting that disables cookie reuse"
+ HTTPRequestDoc.Fields[21].Comments[encoder.LineComment] = "DisableCookie is an optional setting that disables cookie reuse"
+ HTTPRequestDoc.Fields[22].Name = "read-all"
HTTPRequestDoc.Fields[22].Type = "bool"
HTTPRequestDoc.Fields[22].Note = ""
- HTTPRequestDoc.Fields[22].Description = "Redirects specifies whether only redirects to the same host should be followed by the HTTP Client.\n\nThis can be used in conjunction with `max-redirects` to control the HTTP request redirects."
- HTTPRequestDoc.Fields[22].Comments[encoder.LineComment] = "Redirects specifies whether only redirects to the same host should be followed by the HTTP Client."
- HTTPRequestDoc.Fields[23].Name = "pipeline"
+ HTTPRequestDoc.Fields[22].Description = "Enables force reading of the entire raw unsafe request body ignoring\nany specified content length headers."
+ HTTPRequestDoc.Fields[22].Comments[encoder.LineComment] = "Enables force reading of the entire raw unsafe request body ignoring"
+ HTTPRequestDoc.Fields[23].Name = "redirects"
HTTPRequestDoc.Fields[23].Type = "bool"
HTTPRequestDoc.Fields[23].Note = ""
- HTTPRequestDoc.Fields[23].Description = "Pipeline defines if the attack should be performed with HTTP 1.1 Pipelining\n\nAll requests must be idempotent (GET/POST). This can be used for race conditions/billions requests."
- HTTPRequestDoc.Fields[23].Comments[encoder.LineComment] = "Pipeline defines if the attack should be performed with HTTP 1.1 Pipelining"
- HTTPRequestDoc.Fields[24].Name = "unsafe"
+ HTTPRequestDoc.Fields[23].Description = "Redirects specifies whether redirects should be followed by the HTTP Client.\n\nThis can be used in conjunction with `max-redirects` to control the HTTP request redirects."
+ HTTPRequestDoc.Fields[23].Comments[encoder.LineComment] = "Redirects specifies whether redirects should be followed by the HTTP Client."
+ HTTPRequestDoc.Fields[24].Name = "host-redirects"
HTTPRequestDoc.Fields[24].Type = "bool"
HTTPRequestDoc.Fields[24].Note = ""
- HTTPRequestDoc.Fields[24].Description = "Unsafe specifies whether to use rawhttp engine for sending Non RFC-Compliant requests.\n\nThis uses the [rawhttp](https://github.com/projectdiscovery/rawhttp) engine to achieve complete\ncontrol over the request, with no normalization performed by the client."
- HTTPRequestDoc.Fields[24].Comments[encoder.LineComment] = "Unsafe specifies whether to use rawhttp engine for sending Non RFC-Compliant requests."
- HTTPRequestDoc.Fields[25].Name = "race"
+ HTTPRequestDoc.Fields[24].Description = "Redirects specifies whether only redirects to the same host should be followed by the HTTP Client.\n\nThis can be used in conjunction with `max-redirects` to control the HTTP request redirects."
+ HTTPRequestDoc.Fields[24].Comments[encoder.LineComment] = "Redirects specifies whether only redirects to the same host should be followed by the HTTP Client."
+ HTTPRequestDoc.Fields[25].Name = "pipeline"
HTTPRequestDoc.Fields[25].Type = "bool"
HTTPRequestDoc.Fields[25].Note = ""
- HTTPRequestDoc.Fields[25].Description = "Race determines if all the request have to be attempted at the same time (Race Condition)\n\nThe actual number of requests that will be sent is determined by the `race_count` field."
- HTTPRequestDoc.Fields[25].Comments[encoder.LineComment] = "Race determines if all the request have to be attempted at the same time (Race Condition)"
- HTTPRequestDoc.Fields[26].Name = "req-condition"
+ HTTPRequestDoc.Fields[25].Description = "Pipeline defines if the attack should be performed with HTTP 1.1 Pipelining\n\nAll requests must be idempotent (GET/POST). This can be used for race conditions/billions requests."
+ HTTPRequestDoc.Fields[25].Comments[encoder.LineComment] = "Pipeline defines if the attack should be performed with HTTP 1.1 Pipelining"
+ HTTPRequestDoc.Fields[26].Name = "unsafe"
HTTPRequestDoc.Fields[26].Type = "bool"
HTTPRequestDoc.Fields[26].Note = ""
- HTTPRequestDoc.Fields[26].Description = "ReqCondition automatically assigns numbers to requests and preserves their history.\n\nThis allows matching on them later for multi-request conditions."
- HTTPRequestDoc.Fields[26].Comments[encoder.LineComment] = "ReqCondition automatically assigns numbers to requests and preserves their history."
- HTTPRequestDoc.Fields[27].Name = "stop-at-first-match"
+ HTTPRequestDoc.Fields[26].Description = "Unsafe specifies whether to use rawhttp engine for sending Non RFC-Compliant requests.\n\nThis uses the [rawhttp](https://github.com/projectdiscovery/rawhttp) engine to achieve complete\ncontrol over the request, with no normalization performed by the client."
+ HTTPRequestDoc.Fields[26].Comments[encoder.LineComment] = "Unsafe specifies whether to use rawhttp engine for sending Non RFC-Compliant requests."
+ HTTPRequestDoc.Fields[27].Name = "race"
HTTPRequestDoc.Fields[27].Type = "bool"
HTTPRequestDoc.Fields[27].Note = ""
- HTTPRequestDoc.Fields[27].Description = "StopAtFirstMatch stops the execution of the requests and template as soon as a match is found."
- HTTPRequestDoc.Fields[27].Comments[encoder.LineComment] = "StopAtFirstMatch stops the execution of the requests and template as soon as a match is found."
- HTTPRequestDoc.Fields[28].Name = "skip-variables-check"
+ HTTPRequestDoc.Fields[27].Description = "Race determines if all the request have to be attempted at the same time (Race Condition)\n\nThe actual number of requests that will be sent is determined by the `race_count` field."
+ HTTPRequestDoc.Fields[27].Comments[encoder.LineComment] = "Race determines if all the request have to be attempted at the same time (Race Condition)"
+ HTTPRequestDoc.Fields[28].Name = "req-condition"
HTTPRequestDoc.Fields[28].Type = "bool"
HTTPRequestDoc.Fields[28].Note = ""
- HTTPRequestDoc.Fields[28].Description = "SkipVariablesCheck skips the check for unresolved variables in request"
- HTTPRequestDoc.Fields[28].Comments[encoder.LineComment] = "SkipVariablesCheck skips the check for unresolved variables in request"
- HTTPRequestDoc.Fields[29].Name = "iterate-all"
+ HTTPRequestDoc.Fields[28].Description = "ReqCondition automatically assigns numbers to requests and preserves their history.\n\nThis allows matching on them later for multi-request conditions."
+ HTTPRequestDoc.Fields[28].Comments[encoder.LineComment] = "ReqCondition automatically assigns numbers to requests and preserves their history."
+ HTTPRequestDoc.Fields[29].Name = "stop-at-first-match"
HTTPRequestDoc.Fields[29].Type = "bool"
HTTPRequestDoc.Fields[29].Note = ""
- HTTPRequestDoc.Fields[29].Description = "IterateAll iterates all the values extracted from internal extractors"
- HTTPRequestDoc.Fields[29].Comments[encoder.LineComment] = "IterateAll iterates all the values extracted from internal extractors"
- HTTPRequestDoc.Fields[30].Name = "digest-username"
- HTTPRequestDoc.Fields[30].Type = "string"
+ HTTPRequestDoc.Fields[29].Description = "StopAtFirstMatch stops the execution of the requests and template as soon as a match is found."
+ HTTPRequestDoc.Fields[29].Comments[encoder.LineComment] = "StopAtFirstMatch stops the execution of the requests and template as soon as a match is found."
+ HTTPRequestDoc.Fields[30].Name = "skip-variables-check"
+ HTTPRequestDoc.Fields[30].Type = "bool"
HTTPRequestDoc.Fields[30].Note = ""
- HTTPRequestDoc.Fields[30].Description = "DigestAuthUsername specifies the username for digest authentication"
- HTTPRequestDoc.Fields[30].Comments[encoder.LineComment] = "DigestAuthUsername specifies the username for digest authentication"
- HTTPRequestDoc.Fields[31].Name = "digest-password"
- HTTPRequestDoc.Fields[31].Type = "string"
+ HTTPRequestDoc.Fields[30].Description = "SkipVariablesCheck skips the check for unresolved variables in request"
+ HTTPRequestDoc.Fields[30].Comments[encoder.LineComment] = "SkipVariablesCheck skips the check for unresolved variables in request"
+ HTTPRequestDoc.Fields[31].Name = "iterate-all"
+ HTTPRequestDoc.Fields[31].Type = "bool"
HTTPRequestDoc.Fields[31].Note = ""
- HTTPRequestDoc.Fields[31].Description = "DigestAuthPassword specifies the password for digest authentication"
- HTTPRequestDoc.Fields[31].Comments[encoder.LineComment] = "DigestAuthPassword specifies the password for digest authentication"
- HTTPRequestDoc.Fields[32].Name = "disable-path-automerge"
- HTTPRequestDoc.Fields[32].Type = "bool"
+ HTTPRequestDoc.Fields[31].Description = "IterateAll iterates all the values extracted from internal extractors"
+ HTTPRequestDoc.Fields[31].Comments[encoder.LineComment] = "IterateAll iterates all the values extracted from internal extractors"
+ HTTPRequestDoc.Fields[32].Name = "digest-username"
+ HTTPRequestDoc.Fields[32].Type = "string"
HTTPRequestDoc.Fields[32].Note = ""
- HTTPRequestDoc.Fields[32].Description = "DisablePathAutomerge disables merging target url path with raw request path"
- HTTPRequestDoc.Fields[32].Comments[encoder.LineComment] = "DisablePathAutomerge disables merging target url path with raw request path"
- HTTPRequestDoc.Fields[33].Name = "pre-condition"
- HTTPRequestDoc.Fields[33].Type = "[]matchers.Matcher"
+ HTTPRequestDoc.Fields[32].Description = "DigestAuthUsername specifies the username for digest authentication"
+ HTTPRequestDoc.Fields[32].Comments[encoder.LineComment] = "DigestAuthUsername specifies the username for digest authentication"
+ HTTPRequestDoc.Fields[33].Name = "digest-password"
+ HTTPRequestDoc.Fields[33].Type = "string"
HTTPRequestDoc.Fields[33].Note = ""
- HTTPRequestDoc.Fields[33].Description = "Fuzz PreCondition is matcher-like field to check if fuzzing should be performed on this request or not"
- HTTPRequestDoc.Fields[33].Comments[encoder.LineComment] = "Fuzz PreCondition is matcher-like field to check if fuzzing should be performed on this request or not"
- HTTPRequestDoc.Fields[34].Name = "pre-condition-operator"
- HTTPRequestDoc.Fields[34].Type = "string"
+ HTTPRequestDoc.Fields[33].Description = "DigestAuthPassword specifies the password for digest authentication"
+ HTTPRequestDoc.Fields[33].Comments[encoder.LineComment] = "DigestAuthPassword specifies the password for digest authentication"
+ HTTPRequestDoc.Fields[34].Name = "disable-path-automerge"
+ HTTPRequestDoc.Fields[34].Type = "bool"
HTTPRequestDoc.Fields[34].Note = ""
- HTTPRequestDoc.Fields[34].Description = "FuzzPreConditionOperator is the operator between multiple PreConditions for fuzzing Default is OR"
- HTTPRequestDoc.Fields[34].Comments[encoder.LineComment] = "FuzzPreConditionOperator is the operator between multiple PreConditions for fuzzing Default is OR"
+ HTTPRequestDoc.Fields[34].Description = "DisablePathAutomerge disables merging target url path with raw request path"
+ HTTPRequestDoc.Fields[34].Comments[encoder.LineComment] = "DisablePathAutomerge disables merging target url path with raw request path"
+ HTTPRequestDoc.Fields[35].Name = "pre-condition"
+ HTTPRequestDoc.Fields[35].Type = "[]matchers.Matcher"
+ HTTPRequestDoc.Fields[35].Note = ""
+ HTTPRequestDoc.Fields[35].Description = "Fuzz PreCondition is matcher-like field to check if fuzzing should be performed on this request or not"
+ HTTPRequestDoc.Fields[35].Comments[encoder.LineComment] = "Fuzz PreCondition is matcher-like field to check if fuzzing should be performed on this request or not"
+ HTTPRequestDoc.Fields[36].Name = "pre-condition-operator"
+ HTTPRequestDoc.Fields[36].Type = "string"
+ HTTPRequestDoc.Fields[36].Note = ""
+ HTTPRequestDoc.Fields[36].Description = "FuzzPreConditionOperator is the operator between multiple PreConditions for fuzzing Default is OR"
+ HTTPRequestDoc.Fields[36].Comments[encoder.LineComment] = "FuzzPreConditionOperator is the operator between multiple PreConditions for fuzzing Default is OR"
+ HTTPRequestDoc.Fields[37].Name = "global-matchers"
+ HTTPRequestDoc.Fields[37].Type = "bool"
+ HTTPRequestDoc.Fields[37].Note = ""
+ HTTPRequestDoc.Fields[37].Description = "GlobalMatchers marks matchers as static and applies globally to all result events from other templates"
+ HTTPRequestDoc.Fields[37].Comments[encoder.LineComment] = "GlobalMatchers marks matchers as static and applies globally to all result events from other templates"
GENERATORSAttackTypeHolderDoc.Type = "generators.AttackTypeHolder"
GENERATORSAttackTypeHolderDoc.Comments[encoder.LineComment] = " AttackTypeHolder is used to hold internal type of the protocol"
@@ -747,7 +763,7 @@ func init() {
FieldName: "fuzzing",
},
}
- FUZZRuleDoc.Fields = make([]encoder.Doc, 8)
+ FUZZRuleDoc.Fields = make([]encoder.Doc, 9)
FUZZRuleDoc.Fields[0].Name = "type"
FUZZRuleDoc.Fields[0].Type = "string"
FUZZRuleDoc.Fields[0].Note = ""
@@ -762,51 +778,69 @@ func init() {
FUZZRuleDoc.Fields[1].Name = "part"
FUZZRuleDoc.Fields[1].Type = "string"
FUZZRuleDoc.Fields[1].Note = ""
- FUZZRuleDoc.Fields[1].Description = "Part is the part of request to fuzz.\n\nquery fuzzes the query part of url. More parts will be added later."
+ FUZZRuleDoc.Fields[1].Description = "Part is the part of request to fuzz."
FUZZRuleDoc.Fields[1].Comments[encoder.LineComment] = "Part is the part of request to fuzz."
FUZZRuleDoc.Fields[1].Values = []string{
"query",
+ "header",
+ "path",
+ "body",
+ "cookie",
+ "request",
}
- FUZZRuleDoc.Fields[2].Name = "mode"
- FUZZRuleDoc.Fields[2].Type = "string"
+ FUZZRuleDoc.Fields[2].Name = "parts"
+ FUZZRuleDoc.Fields[2].Type = "[]string"
FUZZRuleDoc.Fields[2].Note = ""
- FUZZRuleDoc.Fields[2].Description = "Mode is the mode of fuzzing to perform.\n\nsingle fuzzes one value at a time. multiple fuzzes all values at same time."
- FUZZRuleDoc.Fields[2].Comments[encoder.LineComment] = "Mode is the mode of fuzzing to perform."
+ FUZZRuleDoc.Fields[2].Description = "Parts is the list of parts to fuzz. If multiple parts need to be\ndefined while excluding some, this should be used instead of singular part."
+ FUZZRuleDoc.Fields[2].Comments[encoder.LineComment] = "Parts is the list of parts to fuzz. If multiple parts need to be"
FUZZRuleDoc.Fields[2].Values = []string{
+ "query",
+ "header",
+ "path",
+ "body",
+ "cookie",
+ "request",
+ }
+ FUZZRuleDoc.Fields[3].Name = "mode"
+ FUZZRuleDoc.Fields[3].Type = "string"
+ FUZZRuleDoc.Fields[3].Note = ""
+ FUZZRuleDoc.Fields[3].Description = "Mode is the mode of fuzzing to perform.\n\nsingle fuzzes one value at a time. multiple fuzzes all values at same time."
+ FUZZRuleDoc.Fields[3].Comments[encoder.LineComment] = "Mode is the mode of fuzzing to perform."
+ FUZZRuleDoc.Fields[3].Values = []string{
"single",
"multiple",
}
- FUZZRuleDoc.Fields[3].Name = "keys"
- FUZZRuleDoc.Fields[3].Type = "[]string"
- FUZZRuleDoc.Fields[3].Note = ""
- FUZZRuleDoc.Fields[3].Description = "Keys is the optional list of key named parameters to fuzz."
- FUZZRuleDoc.Fields[3].Comments[encoder.LineComment] = "Keys is the optional list of key named parameters to fuzz."
-
- FUZZRuleDoc.Fields[3].AddExample("Examples of keys", []string{"url", "file", "host"})
- FUZZRuleDoc.Fields[4].Name = "keys-regex"
+ FUZZRuleDoc.Fields[4].Name = "keys"
FUZZRuleDoc.Fields[4].Type = "[]string"
FUZZRuleDoc.Fields[4].Note = ""
- FUZZRuleDoc.Fields[4].Description = "KeysRegex is the optional list of regex key parameters to fuzz."
- FUZZRuleDoc.Fields[4].Comments[encoder.LineComment] = "KeysRegex is the optional list of regex key parameters to fuzz."
+ FUZZRuleDoc.Fields[4].Description = "Keys is the optional list of key named parameters to fuzz."
+ FUZZRuleDoc.Fields[4].Comments[encoder.LineComment] = "Keys is the optional list of key named parameters to fuzz."
- FUZZRuleDoc.Fields[4].AddExample("Examples of key regex", []string{"url.*"})
- FUZZRuleDoc.Fields[5].Name = "values"
+ FUZZRuleDoc.Fields[4].AddExample("Examples of keys", []string{"url", "file", "host"})
+ FUZZRuleDoc.Fields[5].Name = "keys-regex"
FUZZRuleDoc.Fields[5].Type = "[]string"
FUZZRuleDoc.Fields[5].Note = ""
- FUZZRuleDoc.Fields[5].Description = "Values is the optional list of regex value parameters to fuzz."
- FUZZRuleDoc.Fields[5].Comments[encoder.LineComment] = "Values is the optional list of regex value parameters to fuzz."
+ FUZZRuleDoc.Fields[5].Description = "KeysRegex is the optional list of regex key parameters to fuzz."
+ FUZZRuleDoc.Fields[5].Comments[encoder.LineComment] = "KeysRegex is the optional list of regex key parameters to fuzz."
- FUZZRuleDoc.Fields[5].AddExample("Examples of value regex", []string{"https?://.*"})
- FUZZRuleDoc.Fields[6].Name = "fuzz"
- FUZZRuleDoc.Fields[6].Type = "SliceOrMapSlice"
+ FUZZRuleDoc.Fields[5].AddExample("Examples of key regex", []string{"url.*"})
+ FUZZRuleDoc.Fields[6].Name = "values"
+ FUZZRuleDoc.Fields[6].Type = "[]string"
FUZZRuleDoc.Fields[6].Note = ""
- FUZZRuleDoc.Fields[6].Description = "description: |\n Fuzz is the list of payloads to perform substitutions with.\n examples:\n - name: Examples of fuzz\n value: >\n []string{\"{{ssrf}}\", \"{{interactsh-url}}\", \"example-value\"}\n or\n x-header: 1\n x-header: 2"
- FUZZRuleDoc.Fields[6].Comments[encoder.LineComment] = " description: |"
- FUZZRuleDoc.Fields[7].Name = "replace-regex"
- FUZZRuleDoc.Fields[7].Type = "string"
+ FUZZRuleDoc.Fields[6].Description = "Values is the optional list of regex value parameters to fuzz."
+ FUZZRuleDoc.Fields[6].Comments[encoder.LineComment] = "Values is the optional list of regex value parameters to fuzz."
+
+ FUZZRuleDoc.Fields[6].AddExample("Examples of value regex", []string{"https?://.*"})
+ FUZZRuleDoc.Fields[7].Name = "fuzz"
+ FUZZRuleDoc.Fields[7].Type = "SliceOrMapSlice"
FUZZRuleDoc.Fields[7].Note = ""
- FUZZRuleDoc.Fields[7].Description = "replace-regex is regex for regex-replace rule type\nit is only required for replace-regex rule type"
- FUZZRuleDoc.Fields[7].Comments[encoder.LineComment] = "replace-regex is regex for regex-replace rule type"
+ FUZZRuleDoc.Fields[7].Description = "description: |\n Fuzz is the list of payloads to perform substitutions with.\n examples:\n - name: Examples of fuzz\n value: >\n []string{\"{{ssrf}}\", \"{{interactsh-url}}\", \"example-value\"}\n or\n x-header: 1\n x-header: 2"
+ FUZZRuleDoc.Fields[7].Comments[encoder.LineComment] = " description: |"
+ FUZZRuleDoc.Fields[8].Name = "replace-regex"
+ FUZZRuleDoc.Fields[8].Type = "string"
+ FUZZRuleDoc.Fields[8].Note = ""
+ FUZZRuleDoc.Fields[8].Description = "replace-regex is regex for regex-replace rule type\nit is only required for replace-regex rule type"
+ FUZZRuleDoc.Fields[8].Comments[encoder.LineComment] = "replace-regex is regex for regex-replace rule type"
SliceOrMapSliceDoc.Type = "SliceOrMapSlice"
SliceOrMapSliceDoc.Comments[encoder.LineComment] = ""
@@ -819,6 +853,30 @@ func init() {
}
SliceOrMapSliceDoc.Fields = make([]encoder.Doc, 0)
+ ANALYZERSAnalyzerTemplateDoc.Type = "analyzers.AnalyzerTemplate"
+ ANALYZERSAnalyzerTemplateDoc.Comments[encoder.LineComment] = " AnalyzerTemplate is the template for the analyzer"
+ ANALYZERSAnalyzerTemplateDoc.Description = "AnalyzerTemplate is the template for the analyzer"
+ ANALYZERSAnalyzerTemplateDoc.AppearsIn = []encoder.Appearance{
+ {
+ TypeName: "http.Request",
+ FieldName: "analyzer",
+ },
+ }
+ ANALYZERSAnalyzerTemplateDoc.Fields = make([]encoder.Doc, 2)
+ ANALYZERSAnalyzerTemplateDoc.Fields[0].Name = "name"
+ ANALYZERSAnalyzerTemplateDoc.Fields[0].Type = "string"
+ ANALYZERSAnalyzerTemplateDoc.Fields[0].Note = ""
+ ANALYZERSAnalyzerTemplateDoc.Fields[0].Description = "Name is the name of the analyzer to use"
+ ANALYZERSAnalyzerTemplateDoc.Fields[0].Comments[encoder.LineComment] = "Name is the name of the analyzer to use"
+ ANALYZERSAnalyzerTemplateDoc.Fields[0].Values = []string{
+ "time_delay",
+ }
+ ANALYZERSAnalyzerTemplateDoc.Fields[1].Name = "parameters"
+ ANALYZERSAnalyzerTemplateDoc.Fields[1].Type = "map[string]interface{}"
+ ANALYZERSAnalyzerTemplateDoc.Fields[1].Note = ""
+ ANALYZERSAnalyzerTemplateDoc.Fields[1].Description = "Parameters is the parameters for the analyzer\n\nParameters are different for each analyzer. For example, you can customize\ntime_delay analyzer with sleep_duration, time_slope_error_range, etc. Refer\nto the docs for each analyzer to get an idea about parameters."
+ ANALYZERSAnalyzerTemplateDoc.Fields[1].Comments[encoder.LineComment] = "Parameters is the parameters for the analyzer"
+
SignatureTypeHolderDoc.Type = "SignatureTypeHolder"
SignatureTypeHolderDoc.Comments[encoder.LineComment] = " SignatureTypeHolder is used to hold internal type of the signature"
SignatureTypeHolderDoc.Description = "SignatureTypeHolder is used to hold internal type of the signature"
@@ -1306,7 +1364,7 @@ func init() {
Value: "Full Network protocol data",
},
}
- NETWORKRequestDoc.Fields = make([]encoder.Doc, 10)
+ NETWORKRequestDoc.Fields = make([]encoder.Doc, 11)
NETWORKRequestDoc.Fields[0].Name = "id"
NETWORKRequestDoc.Fields[0].Type = "string"
NETWORKRequestDoc.Fields[0].Note = ""
@@ -1365,6 +1423,11 @@ func init() {
NETWORKRequestDoc.Fields[9].Comments[encoder.LineComment] = "ReadAll determines if the data stream should be read till the end regardless of the size"
NETWORKRequestDoc.Fields[9].AddExample("", false)
+ NETWORKRequestDoc.Fields[10].Name = "stop-at-first-match"
+ NETWORKRequestDoc.Fields[10].Type = "bool"
+ NETWORKRequestDoc.Fields[10].Note = ""
+ NETWORKRequestDoc.Fields[10].Description = "StopAtFirstMatch stops the execution of the requests and template as soon as a match is found."
+ NETWORKRequestDoc.Fields[10].Comments[encoder.LineComment] = "StopAtFirstMatch stops the execution of the requests and template as soon as a match is found."
NETWORKInputDoc.Type = "network.Input"
NETWORKInputDoc.Comments[encoder.LineComment] = ""
@@ -1580,7 +1643,12 @@ func init() {
"time",
"select",
"files",
+ "waitdom",
+ "waitfcp",
+ "waitfmp",
+ "waitidle",
"waitload",
+ "waitstable",
"getresource",
"extract",
"setmethod",
@@ -1589,6 +1657,7 @@ func init() {
"deleteheader",
"setbody",
"waitevent",
+ "dialog",
"keyboard",
"debug",
"sleep",
@@ -1627,25 +1696,113 @@ func init() {
},
}
SSLRequestDoc.PartDefinitions = []encoder.KeyValue{
+ {
+ Key: "template-id",
+ Value: "ID of the template executed",
+ },
+ {
+ Key: "template-info",
+ Value: "Info Block of the template executed",
+ },
+ {
+ Key: "template-path",
+ Value: "Path of the template executed",
+ },
+ {
+ Key: "host",
+ Value: "Host is the input to the template",
+ },
+ {
+ Key: "port",
+ Value: "Port is the port of the host",
+ },
+ {
+ Key: "matched",
+ Value: "Matched is the input which was matched upon",
+ },
{
Key: "type",
Value: "Type is the type of request made",
},
+ {
+ Key: "timestamp",
+ Value: "Timestamp is the time when the request was made",
+ },
{
Key: "response",
Value: "JSON SSL protocol handshake details",
},
+ {
+ Key: "cipher",
+ Value: "Cipher is the encryption algorithm used",
+ },
+ {
+ Key: "domains",
+ Value: "Domains are the list of domain names in the certificate",
+ },
+ {
+ Key: "fingerprint_hash",
+ Value: "Fingerprint hash is the unique identifier of the certificate",
+ },
+ {
+ Key: "ip",
+ Value: "IP is the IP address of the server",
+ },
+ {
+ Key: "issuer_cn",
+ Value: "Issuer CN is the common name of the certificate issuer",
+ },
+ {
+ Key: "issuer_dn",
+ Value: "Issuer DN is the distinguished name of the certificate issuer",
+ },
+ {
+ Key: "issuer_org",
+ Value: "Issuer organization is the organization of the certificate issuer",
+ },
{
Key: "not_after",
Value: "Timestamp after which the remote cert expires",
},
{
- Key: "host",
- Value: "Host is the input to the template",
+ Key: "not_before",
+ Value: "Timestamp before which the certificate is not valid",
},
{
- Key: "matched",
- Value: "Matched is the input which was matched upon",
+ Key: "probe_status",
+ Value: "Probe status indicates if the probe was successful",
+ },
+ {
+ Key: "serial",
+ Value: "Serial is the serial number of the certificate",
+ },
+ {
+ Key: "sni",
+ Value: "SNI is the server name indication used in the handshake",
+ },
+ {
+ Key: "subject_an",
+ Value: "Subject AN is the list of subject alternative names",
+ },
+ {
+ Key: "subject_cn",
+ Value: "Subject CN is the common name of the certificate subject",
+ },
+ {
+ Key: "subject_dn",
+ Value: "Subject DN is the distinguished name of the certificate subject",
+ },
+ {
+ Key: "subject_org",
+ Value: "Subject organization is the organization of the certificate subject",
+ },
+ {
+ Key: "tls_connection",
+ Value: "TLS connection is the type of TLS connection used",
+ },
+ {
+ Key: "tls_version",
+ Value: "TLS version is the version of the TLS protocol used",
},
}
SSLRequestDoc.Fields = make([]encoder.Doc, 9)
@@ -1911,7 +2068,7 @@ func init() {
Value: "Matched is the input which was matched upon",
},
}
- JAVASCRIPTRequestDoc.Fields = make([]encoder.Doc, 10)
+ JAVASCRIPTRequestDoc.Fields = make([]encoder.Doc, 9)
JAVASCRIPTRequestDoc.Fields[0].Name = "id"
JAVASCRIPTRequestDoc.Fields[0].Type = "string"
JAVASCRIPTRequestDoc.Fields[0].Note = ""
@@ -1937,33 +2094,28 @@ func init() {
JAVASCRIPTRequestDoc.Fields[4].Note = ""
JAVASCRIPTRequestDoc.Fields[4].Description = "Code contains code to execute for the javascript request."
JAVASCRIPTRequestDoc.Fields[4].Comments[encoder.LineComment] = "Code contains code to execute for the javascript request."
- JAVASCRIPTRequestDoc.Fields[5].Name = "timeout"
- JAVASCRIPTRequestDoc.Fields[5].Type = "int"
+ JAVASCRIPTRequestDoc.Fields[5].Name = "stop-at-first-match"
+ JAVASCRIPTRequestDoc.Fields[5].Type = "bool"
JAVASCRIPTRequestDoc.Fields[5].Note = ""
- JAVASCRIPTRequestDoc.Fields[5].Description = "Timeout in seconds is optional timeout for each javascript script execution (i.e init, pre-condition, code)"
- JAVASCRIPTRequestDoc.Fields[5].Comments[encoder.LineComment] = "Timeout in seconds is optional timeout for each javascript script execution (i.e init, pre-condition, code)"
- JAVASCRIPTRequestDoc.Fields[6].Name = "stop-at-first-match"
- JAVASCRIPTRequestDoc.Fields[6].Type = "bool"
+ JAVASCRIPTRequestDoc.Fields[5].Description = "StopAtFirstMatch stops processing the request at first match."
+ JAVASCRIPTRequestDoc.Fields[5].Comments[encoder.LineComment] = "StopAtFirstMatch stops processing the request at first match."
+ JAVASCRIPTRequestDoc.Fields[6].Name = "attack"
+ JAVASCRIPTRequestDoc.Fields[6].Type = "generators.AttackTypeHolder"
JAVASCRIPTRequestDoc.Fields[6].Note = ""
- JAVASCRIPTRequestDoc.Fields[6].Description = "StopAtFirstMatch stops processing the request at first match."
- JAVASCRIPTRequestDoc.Fields[6].Comments[encoder.LineComment] = "StopAtFirstMatch stops processing the request at first match."
- JAVASCRIPTRequestDoc.Fields[7].Name = "attack"
- JAVASCRIPTRequestDoc.Fields[7].Type = "generators.AttackTypeHolder"
+ JAVASCRIPTRequestDoc.Fields[6].Description = "Attack is the type of payload combinations to perform.\n\nSniper is each payload once, pitchfork combines multiple payload sets and clusterbomb generates\npermutations and combinations for all payloads."
+ JAVASCRIPTRequestDoc.Fields[6].Comments[encoder.LineComment] = "Attack is the type of payload combinations to perform."
+ JAVASCRIPTRequestDoc.Fields[7].Name = "threads"
+ JAVASCRIPTRequestDoc.Fields[7].Type = "int"
JAVASCRIPTRequestDoc.Fields[7].Note = ""
- JAVASCRIPTRequestDoc.Fields[7].Description = "Attack is the type of payload combinations to perform.\n\nSniper is each payload once, pitchfork combines multiple payload sets and clusterbomb generates\npermutations and combinations for all payloads."
- JAVASCRIPTRequestDoc.Fields[7].Comments[encoder.LineComment] = "Attack is the type of payload combinations to perform."
- JAVASCRIPTRequestDoc.Fields[8].Name = "threads"
- JAVASCRIPTRequestDoc.Fields[8].Type = "int"
- JAVASCRIPTRequestDoc.Fields[8].Note = ""
- JAVASCRIPTRequestDoc.Fields[8].Description = "Payload concurreny i.e threads for sending requests."
- JAVASCRIPTRequestDoc.Fields[8].Comments[encoder.LineComment] = "Payload concurreny i.e threads for sending requests."
+ JAVASCRIPTRequestDoc.Fields[7].Description = "Payload concurreny i.e threads for sending requests."
+ JAVASCRIPTRequestDoc.Fields[7].Comments[encoder.LineComment] = "Payload concurreny i.e threads for sending requests."
- JAVASCRIPTRequestDoc.Fields[8].AddExample("Send requests using 10 concurrent threads", 10)
- JAVASCRIPTRequestDoc.Fields[9].Name = "payloads"
- JAVASCRIPTRequestDoc.Fields[9].Type = "map[string]interface{}"
- JAVASCRIPTRequestDoc.Fields[9].Note = ""
- JAVASCRIPTRequestDoc.Fields[9].Description = "Payloads contains any payloads for the current request.\n\nPayloads support both key-values combinations where a list\nof payloads is provided, or optionally a single file can also\nbe provided as payload which will be read on run-time."
- JAVASCRIPTRequestDoc.Fields[9].Comments[encoder.LineComment] = "Payloads contains any payloads for the current request."
+ JAVASCRIPTRequestDoc.Fields[7].AddExample("Send requests using 10 concurrent threads", 10)
+ JAVASCRIPTRequestDoc.Fields[8].Name = "payloads"
+ JAVASCRIPTRequestDoc.Fields[8].Type = "map[string]interface{}"
+ JAVASCRIPTRequestDoc.Fields[8].Note = ""
+ JAVASCRIPTRequestDoc.Fields[8].Description = "Payloads contains any payloads for the current request.\n\nPayloads support both key-values combinations where a list\nof payloads is provided, or optionally a single file can also\nbe provided as payload which will be read on run-time."
+ JAVASCRIPTRequestDoc.Fields[8].Comments[encoder.LineComment] = "Payloads contains any payloads for the current request."
HTTPSignatureTypeHolderDoc.Type = "http.SignatureTypeHolder"
HTTPSignatureTypeHolderDoc.Comments[encoder.LineComment] = " SignatureTypeHolder is used to hold internal type of the signature"
@@ -2005,6 +2157,7 @@ func GetTemplateDoc() *encoder.FileDoc {
&HTTPMethodTypeHolderDoc,
&FUZZRuleDoc,
&SliceOrMapSliceDoc,
+ &ANALYZERSAnalyzerTemplateDoc,
&SignatureTypeHolderDoc,
&MATCHERSMatcherDoc,
&MatcherTypeHolderDoc,
diff --git a/pkg/templates/templates_doc_examples.go b/pkg/templates/templates_doc_examples.go
index 7ae5b081c6..1ed0f93c07 100644
--- a/pkg/templates/templates_doc_examples.go
+++ b/pkg/templates/templates_doc_examples.go
@@ -3,6 +3,7 @@
package templates
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
@@ -13,6 +14,31 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/protocols/file"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/http"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/network"
+=======
+<<<<<<< HEAD:v2/pkg/templates/templates_doc_examples.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/extractors"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators/matchers"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/dns"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/file"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/http"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/network"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/dns"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/file"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/http"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/network"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/templates/templates_doc_examples.go
+>>>>>>> projectdiscovery-main
)
var (
diff --git a/pkg/templates/tests/workflow-invalid.yaml b/pkg/templates/tests/workflow-invalid.yaml
new file mode 100644
index 0000000000..981823aa0c
--- /dev/null
+++ b/pkg/templates/tests/workflow-invalid.yaml
@@ -0,0 +1,18 @@
+id: workflow-example
+
+info:
+ name: Test Invalid Workflow Template
+ author: pdteam
+ severity: info
+
+http:
+ - raw:
+ - |
+ POST /re HTTP/1.1
+ Host: {{Hostname}}
+
+ {{code_response}}
+
+workflows:
+ - template: tests/match-1.yaml
+ - template: tests/match-1.yaml
diff --git a/pkg/templates/types/types.go b/pkg/templates/types/types.go
index e80b45d776..302cf70267 100644
--- a/pkg/templates/types/types.go
+++ b/pkg/templates/types/types.go
@@ -9,6 +9,14 @@ import (
"github.com/invopop/jsonschema"
"github.com/pkg/errors"
"github.com/projectdiscovery/goflags"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/templates/types/types.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/templates/types/types.go
+>>>>>>> projectdiscovery-main
)
// ProtocolType is the type of the request protocol specified
@@ -69,6 +77,18 @@ func GetSupportedProtocolTypes() ProtocolTypes {
return result
}
+// SupportedProtocolsStrings returns a slice of strings of supported protocols
+func SupportedProtocolsStrings() []string {
+ var result []string
+ for _, protocol := range GetSupportedProtocolTypes() {
+ if protocol.String() == "" {
+ continue
+ }
+ result = append(result, protocol.String())
+ }
+ return result
+}
+
func toProtocolType(valueToMap string) (ProtocolType, error) {
normalizedValue := normalizeValue(valueToMap)
for key, currentValue := range protocolMappings {
diff --git a/pkg/templates/workflows.go b/pkg/templates/workflows.go
index ce98481919..64e62037ec 100644
--- a/pkg/templates/workflows.go
+++ b/pkg/templates/workflows.go
@@ -8,6 +8,20 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/utils/stats"
"github.com/Explorer1092/nuclei/v3/pkg/workflows"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/templates/workflows.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/workflows"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/keys"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils/stats"
+ "github.com/projectdiscovery/nuclei/v3/pkg/workflows"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/templates/workflows.go
+>>>>>>> projectdiscovery-main
)
// compileWorkflow compiles the workflow for execution
@@ -105,7 +119,7 @@ func parseWorkflowTemplate(workflow *workflows.WorkflowTemplate, preprocessor Pr
// increment signed/unsigned counters
if template.Verified {
if template.TemplateVerifier == "" {
- SignatureStats[PDVerifier].Add(1)
+ SignatureStats[keys.PDVerifier].Add(1)
} else {
SignatureStats[template.TemplateVerifier].Add(1)
}
diff --git a/pkg/testutils/fuzzplayground/server.go b/pkg/testutils/fuzzplayground/server.go
index 04f00a99df..af42552cbf 100644
--- a/pkg/testutils/fuzzplayground/server.go
+++ b/pkg/testutils/fuzzplayground/server.go
@@ -27,6 +27,7 @@ func GetPlaygroundServer() *echo.Echo {
e.GET("/request", requestHandler)
e.GET("/email", emailHandler)
e.GET("/permissions", permissionsHandler)
+
e.GET("/blog/post", numIdorHandler) // for num based idors like ?id=44
e.POST("/reset-password", resetPasswordHandler)
e.GET("/host-header-lab", hostHeaderLabHandler)
@@ -47,13 +48,20 @@ var bodyTemplate = `
func indexHandler(ctx echo.Context) error {
return ctx.HTML(200, fmt.Sprintf(bodyTemplate, `Fuzzing Playground
-
+
`))
}
diff --git a/pkg/testutils/integration.go b/pkg/testutils/integration.go
index 8b5a71b1ab..49eda29626 100644
--- a/pkg/testutils/integration.go
+++ b/pkg/testutils/integration.go
@@ -14,6 +14,7 @@ import (
"github.com/gobwas/ws"
"github.com/julienschmidt/httprouter"
+ "github.com/projectdiscovery/utils/conversion"
)
// ExtraArgs
@@ -70,20 +71,26 @@ func RunNucleiBareArgsAndGetResults(debug bool, env []string, extra ...string) (
} else {
cmd.Args = append(cmd.Args, "-silent")
}
- data, err := cmd.Output()
- if debug {
- fmt.Println(string(data))
+ output, err := cmd.Output()
+ var data string
+ if len(output) > 0 {
+ data = strings.TrimSpace(conversion.String(output))
}
- if len(data) < 1 && err != nil {
- return nil, fmt.Errorf("%v: %v", err.Error(), string(data))
+ if debug {
+ fmt.Println(data)
}
var parts []string
- items := strings.Split(string(data), "\n")
+ items := strings.Split(data, "\n")
for _, i := range items {
if i != "" {
parts = append(parts, i)
}
}
+
+ if (data == "" || len(parts) == 0) && err != nil {
+ return nil, fmt.Errorf("%v: %v", err.Error(), data)
+ }
+
return parts, nil
}
@@ -98,20 +105,25 @@ func RunNucleiWithArgsAndGetResults(debug bool, args ...string) ([]string, error
} else {
cmd.Args = append(cmd.Args, "-silent")
}
- data, err := cmd.Output()
- if debug {
- fmt.Println(string(data))
+ output, err := cmd.Output()
+ var data string
+ if len(output) > 0 {
+ data = strings.TrimSpace(conversion.String(output))
}
- if len(data) < 1 && err != nil {
- return nil, fmt.Errorf("%v: %v", err.Error(), string(data))
+ if debug {
+ fmt.Println(data)
}
var parts []string
- items := strings.Split(string(data), "\n")
+ items := strings.Split(data, "\n")
for _, i := range items {
if i != "" {
parts = append(parts, i)
}
}
+
+ if (data == "" || len(parts) == 0) && err != nil {
+ return nil, fmt.Errorf("%v: %v", err.Error(), data)
+ }
return parts, nil
}
@@ -127,12 +139,16 @@ func RunNucleiArgsAndGetErrors(debug bool, env []string, extra ...string) ([]str
cmd.Args = append(cmd.Args, "-allow-local-file-access")
cmd.Args = append(cmd.Args, "-nc") // disable color
cmd.Env = append(cmd.Env, ExtraEnvVars...)
- data, err := cmd.CombinedOutput()
+ dataOutput, err := cmd.CombinedOutput()
if debug {
- fmt.Println(string(data))
+ fmt.Println(string(dataOutput))
+ }
+ var data string
+ if len(dataOutput) > 0 {
+ data = strings.TrimSpace(conversion.String(dataOutput))
}
results := []string{}
- for _, v := range strings.Split(string(data), "\n") {
+ for _, v := range strings.Split(data, "\n") {
line := strings.TrimSpace(v)
switch {
case strings.HasPrefix(line, "[ERR]"):
@@ -146,7 +162,7 @@ func RunNucleiArgsAndGetErrors(debug bool, env []string, extra ...string) ([]str
return results, err
}
-// RunNucleiArgsWithEnvAndGetErrors returns a list of errors in nuclei output (ERR,WRN,FTL)
+// RunNucleiArgsWithEnvAndGetResults returns a list of results in nuclei output (ERR,WRN,FTL)
func RunNucleiArgsWithEnvAndGetResults(debug bool, env []string, extra ...string) ([]string, error) {
cmd := exec.Command("./nuclei")
extra = append(extra, ExtraDebugArgs...)
@@ -164,20 +180,24 @@ func RunNucleiArgsWithEnvAndGetResults(debug bool, env []string, extra ...string
} else {
cmd.Args = append(cmd.Args, "-silent")
}
- data, err := cmd.Output()
- if debug {
- fmt.Println(string(data))
+ dataOutput, err := cmd.Output()
+ var data string
+ if len(dataOutput) > 0 {
+ data = strings.TrimSpace(conversion.String(dataOutput))
}
- if len(data) < 1 && err != nil {
- return nil, fmt.Errorf("%v: %v", err.Error(), string(data))
+ if debug {
+ fmt.Println(data)
}
var parts []string
- items := strings.Split(string(data), "\n")
+ items := strings.Split(data, "\n")
for _, i := range items {
if i != "" {
parts = append(parts, i)
}
}
+ if (data == "" || len(parts) == 0) && err != nil {
+ return nil, fmt.Errorf("%v: %v", err.Error(), data)
+ }
return parts, nil
}
diff --git a/pkg/testutils/testutils.go b/pkg/testutils/testutils.go
index f6cc0bd8ad..5dbd7892fb 100644
--- a/pkg/testutils/testutils.go
+++ b/pkg/testutils/testutils.go
@@ -23,10 +23,39 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/types"
"github.com/Explorer1092/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/gologger/levels"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/testutils/testutils.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/disk"
+ "github.com/Explorer1092/nuclei/v2/pkg/model"
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/output"
+ "github.com/Explorer1092/nuclei/v2/pkg/progress"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols/common/protocolinit"
+ "github.com/Explorer1092/nuclei/v2/pkg/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/progress"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
+ protocolUtils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+ "github.com/projectdiscovery/nuclei/v3/pkg/utils"
+ unitutils "github.com/projectdiscovery/utils/unit"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/testutils/testutils.go
+>>>>>>> projectdiscovery-main
)
// Init initializes the protocols and their configurations
func Init(options *types.Options) {
+ _ = protocolstate.Init(options)
_ = protocolinit.Init(options)
}
@@ -73,7 +102,6 @@ var DefaultOptions = &types.Options{
InteractionsPollDuration: 5,
GitHubTemplateRepo: []string{},
GitHubToken: "",
- ResponseReadTimeout: time.Second * 5,
}
// TemplateInfo contains info for a mock executed template.
@@ -87,17 +115,17 @@ type TemplateInfo struct {
func NewMockExecuterOptions(options *types.Options, info *TemplateInfo) *protocols.ExecutorOptions {
progressImpl, _ := progress.NewStatsTicker(0, false, false, false, 0)
executerOpts := &protocols.ExecutorOptions{
- TemplateID: info.ID,
- TemplateInfo: info.Info,
- TemplatePath: info.Path,
- Output: NewMockOutputWriter(options.OmitTemplate),
- Options: options,
- Progress: progressImpl,
- ProjectFile: nil,
- IssuesClient: nil,
- Browser: nil,
- Catalog: disk.NewCatalog(config.DefaultConfig.TemplatesDirectory),
- RateLimiter: ratelimit.New(context.Background(), uint(options.RateLimit), time.Second),
+ TemplateID: info.ID,
+ TemplateInfo: info.Info,
+ TemplatePath: info.Path,
+ Output: NewMockOutputWriter(options.OmitTemplate),
+ Options: options,
+ Progress: progressImpl,
+ ProjectFile: nil,
+ IssuesClient: nil,
+ Browser: nil,
+ Catalog: disk.NewCatalog(config.DefaultConfig.TemplatesDirectory),
+ RateLimiter: ratelimit.New(context.Background(), uint(options.RateLimit), time.Second),
}
executerOpts.CreateTemplateCtxStore()
return executerOpts
@@ -165,7 +193,7 @@ func (m *MockOutputWriter) WriteFailure(wrappedEvent *output.InternalWrappedEven
// create event
event := wrappedEvent.InternalEvent
- templatePath, templateURL := utils.TemplatePathURL(types.ToString(event["template-path"]), types.ToString(event["template-id"]))
+ templatePath, templateURL := utils.TemplatePathURL(types.ToString(event["template-path"]), types.ToString(event["template-id"]), types.ToString(event["template-verifier"]))
var templateInfo model.Info
if ti, ok := event["template-info"].(model.Info); ok {
templateInfo = ti
@@ -201,7 +229,7 @@ func (m *MockOutputWriter) WriteFailure(wrappedEvent *output.InternalWrappedEven
return m.Write(data)
}
-var maxTemplateFileSizeForEncoding = 1024 * 1024
+var maxTemplateFileSizeForEncoding = unitutils.Mega
func (w *MockOutputWriter) encodeTemplate(templatePath string) string {
data, err := os.ReadFile(templatePath)
diff --git a/pkg/tmplexec/exec.go b/pkg/tmplexec/exec.go
index 9a9f7d12d0..55b19a545b 100644
--- a/pkg/tmplexec/exec.go
+++ b/pkg/tmplexec/exec.go
@@ -19,6 +19,21 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/tmplexec/multiproto"
"github.com/dop251/goja"
"github.com/projectdiscovery/gologger"
+<<<<<<< HEAD
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/js/compiler"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/writer"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan/events"
+ "github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/flow"
+ "github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/generic"
+ "github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/multiproto"
+ "github.com/projectdiscovery/utils/errkit"
+>>>>>>> projectdiscovery-main
)
// TemplateExecutor is an executor for a template
@@ -68,8 +83,10 @@ func (e *TemplateExecuter) Compile() error {
if cliOptions.Verbose {
rawErrorMessage := dslCompilationError.Error()
formattedErrorMessage := strings.ToUpper(rawErrorMessage[:1]) + rawErrorMessage[1:] + "."
- gologger.Warning().Msgf(formattedErrorMessage)
+
+ gologger.Warning().Msg(formattedErrorMessage)
gologger.Info().Msgf("The available custom DSL functions are:")
+
fmt.Println(dsl.GetPrintableDslFunctionSignatures(cliOptions.NoColor))
}
}
@@ -124,6 +141,8 @@ func (e *TemplateExecuter) Execute(ctx *scan.ScanContext) (bool, error) {
executed := &atomic.Bool{}
// matched in this case means something was exported / written to output
matched := &atomic.Bool{}
+ // callbackCalled tracks if the callback was called or not
+ callbackCalled := &atomic.Bool{}
defer func() {
// it is essential to remove template context of `Scan i.e template x input pair`
// since it is of no use after scan is completed (regardless of success or failure)
@@ -141,6 +160,7 @@ func (e *TemplateExecuter) Execute(ctx *scan.ScanContext) (bool, error) {
}
ctx.OnResult = func(event *output.InternalWrappedEvent) {
+ callbackCalled.Store(true)
if event == nil {
// something went wrong
return
@@ -167,7 +187,15 @@ func (e *TemplateExecuter) Execute(ctx *scan.ScanContext) (bool, error) {
if !event.HasOperatorResult() && event.InternalEvent != nil {
lastMatcherEvent = event
} else {
- if writer.WriteResult(event, e.options.Output, e.options.Progress, e.options.IssuesClient) {
+ var isGlobalMatchers bool
+ isGlobalMatchers, _ = event.InternalEvent["global-matchers"].(bool)
+ // NOTE(dwisiswant0): Don't store `matched` on a `global-matchers` template.
+ // This will end up generating 2 events from the same `scan.ScanContext` if
+ // one of the templates has `global-matchers` enabled. This way,
+ // non-`global-matchers` templates can enter the `writeFailureCallback`
+ // func to log failure output.
+ wr := writer.WriteResult(event, e.options.Output, e.options.Progress, e.options.IssuesClient)
+ if wr && !isGlobalMatchers {
matched.Store(true)
} else {
lastMatcherEvent = event
@@ -196,13 +224,62 @@ func (e *TemplateExecuter) Execute(ctx *scan.ScanContext) (bool, error) {
} else {
errx = e.engine.ExecuteWithResults(ctx)
}
+ ctx.LogError(errx)
if lastMatcherEvent != nil {
+ lastMatcherEvent.Lock()
+ lastMatcherEvent.InternalEvent["error"] = getErrorCause(ctx.GenerateErrorMessage())
+ lastMatcherEvent.Unlock()
writeFailureCallback(lastMatcherEvent, e.options.Options.MatcherStatus)
}
+
+ //TODO: this is a hacky way to handle the case where the callback is not called and matcher-status is true.
+ // This is a workaround and needs to be refactored.
+ // Check if callback was never called and matcher-status is true
+ if !callbackCalled.Load() && e.options.Options.MatcherStatus {
+ fakeEvent := &output.InternalWrappedEvent{
+ Results: []*output.ResultEvent{
+ {
+ TemplateID: e.options.TemplateID,
+ Info: e.options.TemplateInfo,
+ Type: e.getTemplateType(),
+ Host: ctx.Input.MetaInput.Input,
+ Error: getErrorCause(ctx.GenerateErrorMessage()),
+ },
+ },
+ OperatorsResult: &operators.Result{
+ Matched: false,
+ },
+ }
+ writeFailureCallback(fakeEvent, e.options.Options.MatcherStatus)
+ }
+
return executed.Load() || matched.Load(), errx
}
+// getErrorCause tries to parse the cause of given error
+// this is legacy support due to use of errorutil in existing libraries
+// but this should not be required once all libraries are updated
+func getErrorCause(err error) string {
+ if err == nil {
+ return ""
+ }
+ errx := errkit.FromError(err)
+ var cause error
+ for _, e := range errx.Errors() {
+ if e != nil && strings.Contains(e.Error(), "context deadline exceeded") {
+ continue
+ }
+ cause = e
+ break
+ }
+ if cause == nil {
+ cause = errkit.Append(errkit.New("could not get error cause"), errx)
+ }
+ // parseScanError prettifies the error message and removes everything except the cause
+ return parseScanError(cause.Error())
+}
+
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (e *TemplateExecuter) ExecuteWithResults(ctx *scan.ScanContext) ([]*output.ResultEvent, error) {
var errx error
diff --git a/pkg/tmplexec/flow/flow_executor.go b/pkg/tmplexec/flow/flow_executor.go
index 03f363dc9d..8433708449 100644
--- a/pkg/tmplexec/flow/flow_executor.go
+++ b/pkg/tmplexec/flow/flow_executor.go
@@ -13,7 +13,12 @@ import (
templateTypes "github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/dop251/goja"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/types"
+=======
+ "github.com/kitabisa/go-ci"
+ "github.com/projectdiscovery/nuclei/v3/pkg/types"
+>>>>>>> projectdiscovery-main
errorutil "github.com/projectdiscovery/utils/errors"
fileutil "github.com/projectdiscovery/utils/file"
mapsutil "github.com/projectdiscovery/utils/maps"
@@ -201,7 +206,13 @@ func (f *FlowExecutor) ExecuteWithResults(ctx *scan.ScanContext) error {
}
}()
+
+ // TODO(dwisiswant0): remove this once we get the RCA.
defer func() {
+ if ci.IsCI() {
+ return
+ }
+
if r := recover(); r != nil {
f.ctx.LogError(fmt.Errorf("panic occurred while executing flow: %v", r))
}
diff --git a/pkg/tmplexec/flow/flow_executor_test.go b/pkg/tmplexec/flow/flow_executor_test.go
index 135b8befc5..59c6862af8 100644
--- a/pkg/tmplexec/flow/flow_executor_test.go
+++ b/pkg/tmplexec/flow/flow_executor_test.go
@@ -137,7 +137,7 @@ func TestFlowWithConditionPositive(t *testing.T) {
err = Template.Executer.Compile()
require.Nil(t, err, "could not compile template")
- input := contextargs.NewWithInput(context.Background(), "blog.projectdiscovery.io")
+ input := contextargs.NewWithInput(context.Background(), "cloud.projectdiscovery.io")
ctx := scan.NewScanContext(context.Background(), input)
// positive match . expect results also verify that both dns() and http() were executed
gotresults, err := Template.Executer.Execute(ctx)
@@ -146,11 +146,11 @@ func TestFlowWithConditionPositive(t *testing.T) {
}
func TestFlowWithNoMatchers(t *testing.T) {
+ setup()
// when using conditional flow with no matchers at all
// we implicitly assume that request was successful and internally changed the result to true (for scope of condition only)
- // testcase-1 : no matchers but contains extractor
- Template, err := templates.Parse("testcases/condition-flow-extractors.yaml", nil, executerOpts)
+ Template, err := templates.Parse("testcases/condition-flow-no-operators.yaml", nil, executerOpts)
require.Nil(t, err, "could not parse template")
require.True(t, Template.Flow != "", "not a flow template") // this is classifer if template is flow or not
@@ -158,27 +158,27 @@ func TestFlowWithNoMatchers(t *testing.T) {
err = Template.Executer.Compile()
require.Nil(t, err, "could not compile template")
- input := contextargs.NewWithInput(context.Background(), "blog.projectdiscovery.io")
- ctx := scan.NewScanContext(context.Background(), input)
+ anotherInput := contextargs.NewWithInput(context.Background(), "cloud.projectdiscovery.io")
+ anotherCtx := scan.NewScanContext(context.Background(), anotherInput)
// positive match . expect results also verify that both dns() and http() were executed
- gotresults, err := Template.Executer.Execute(ctx)
+ gotresults, err := Template.Executer.Execute(anotherCtx)
require.Nil(t, err, "could not execute template")
require.True(t, gotresults)
- // testcase-2 : no matchers and no extractors
- Template, err = templates.Parse("testcases/condition-flow-no-operators.yaml", nil, executerOpts)
- require.Nil(t, err, "could not parse template")
+ t.Run("Contains Extractor", func(t *testing.T) {
+ Template, err := templates.Parse("testcases/condition-flow-extractors.yaml", nil, executerOpts)
+ require.Nil(t, err, "could not parse template")
- require.True(t, Template.Flow != "", "not a flow template") // this is classifer if template is flow or not
+ require.True(t, Template.Flow != "", "not a flow template") // this is classifer if template is flow or not
- err = Template.Executer.Compile()
- require.Nil(t, err, "could not compile template")
-
- anotherInput := contextargs.NewWithInput(context.Background(), "blog.projectdiscovery.io")
- anotherCtx := scan.NewScanContext(context.Background(), anotherInput)
- // positive match . expect results also verify that both dns() and http() were executed
- gotresults, err = Template.Executer.Execute(anotherCtx)
- require.Nil(t, err, "could not execute template")
- require.True(t, gotresults)
+ err = Template.Executer.Compile()
+ require.Nil(t, err, "could not compile template")
+ input := contextargs.NewWithInput(context.Background(), "scanme.sh")
+ ctx := scan.NewScanContext(context.Background(), input)
+ // positive match . expect results also verify that both dns() and http() were executed
+ gotresults, err := Template.Executer.Execute(ctx)
+ require.Nil(t, err, "could not execute template")
+ require.True(t, gotresults)
+ })
}
diff --git a/pkg/tmplexec/flow/flow_internal.go b/pkg/tmplexec/flow/flow_internal.go
index 2968c88ccf..8a2f12dbc0 100644
--- a/pkg/tmplexec/flow/flow_internal.go
+++ b/pkg/tmplexec/flow/flow_internal.go
@@ -29,7 +29,15 @@ func (f *FlowExecutor) requestExecutor(runtime *goja.Runtime, reqMap mapsutil.Ma
// execution logic for http()/dns() etc
for index := range f.allProtocols[opts.protoName] {
req := f.allProtocols[opts.protoName][index]
- err := req.ExecuteWithResults(f.ctx.Input, output.InternalEvent(f.options.GetTemplateCtx(f.ctx.Input.MetaInput).GetAll()), nil, f.protocolResultCallback(req, matcherStatus, opts))
+ // transform input if required
+ inputItem := f.ctx.Input.Clone()
+ if f.options.InputHelper != nil && f.ctx.Input.MetaInput.Input != "" {
+ if inputItem.MetaInput.Input = f.options.InputHelper.Transform(inputItem.MetaInput.Input, req.Type()); inputItem.MetaInput.Input == "" {
+ f.ctx.LogError(fmt.Errorf("failed to transform input for protocol %s", req.Type()))
+ return false
+ }
+ }
+ err := req.ExecuteWithResults(inputItem, output.InternalEvent(f.options.GetTemplateCtx(f.ctx.Input.MetaInput).GetAll()), nil, f.protocolResultCallback(req, matcherStatus, opts))
if err != nil {
// save all errors in a map with id as key
// its less likely that there will be race condition but just in case
@@ -58,7 +66,15 @@ func (f *FlowExecutor) requestExecutor(runtime *goja.Runtime, reqMap mapsutil.Ma
}
return matcherStatus.Load()
}
- err := req.ExecuteWithResults(f.ctx.Input, output.InternalEvent(f.options.GetTemplateCtx(f.ctx.Input.MetaInput).GetAll()), nil, f.protocolResultCallback(req, matcherStatus, opts))
+ // transform input if required
+ inputItem := f.ctx.Input.Clone()
+ if f.options.InputHelper != nil && f.ctx.Input.MetaInput.Input != "" {
+ if inputItem.MetaInput.Input = f.options.InputHelper.Transform(inputItem.MetaInput.Input, req.Type()); inputItem.MetaInput.Input == "" {
+ f.ctx.LogError(fmt.Errorf("failed to transform input for protocol %s", req.Type()))
+ return false
+ }
+ }
+ err := req.ExecuteWithResults(inputItem, output.InternalEvent(f.options.GetTemplateCtx(f.ctx.Input.MetaInput).GetAll()), nil, f.protocolResultCallback(req, matcherStatus, opts))
if err != nil {
index := id
err = f.allErrs.Set(opts.protoName+":"+index, err)
@@ -72,7 +88,7 @@ func (f *FlowExecutor) requestExecutor(runtime *goja.Runtime, reqMap mapsutil.Ma
// protocolResultCallback returns a callback that is executed
// after execution of each protocol request
-func (f *FlowExecutor) protocolResultCallback(req protocols.Request, matcherStatus *atomic.Bool, opts *ProtoOptions) func(result *output.InternalWrappedEvent) {
+func (f *FlowExecutor) protocolResultCallback(req protocols.Request, matcherStatus *atomic.Bool, _ *ProtoOptions) func(result *output.InternalWrappedEvent) {
return func(result *output.InternalWrappedEvent) {
if result != nil {
// Note: flow specific implicit behaviours should be handled here
diff --git a/pkg/tmplexec/flow/testcases/condition-flow-extractors.yaml b/pkg/tmplexec/flow/testcases/condition-flow-extractors.yaml
index 8dcb7c4f06..a1fba0dba8 100644
--- a/pkg/tmplexec/flow/testcases/condition-flow-extractors.yaml
+++ b/pkg/tmplexec/flow/testcases/condition-flow-extractors.yaml
@@ -1,29 +1,28 @@
-id: ghost-blog-detection
+id: condition-flow-extractors
info:
- name: Ghost blog detection
+ name: Condition Flow Extractors
author: pdteam
severity: info
-
flow: dns() && http()
dns:
- name: "{{FQDN}}"
- type: CNAME
+ type: A
extractors:
- type: dsl
- name: cname
+ name: a
internal: true
dsl:
- - cname
-
+ - a
+
http:
- method: GET
path:
- - "{{BaseURL}}?ref={{cname}}"
+ - "{{BaseURL}}/?ref={{a}}"
matchers:
- type: word
words:
- - "ghost.io"
\ No newline at end of file
+ - "ok"
\ No newline at end of file
diff --git a/pkg/tmplexec/flow/testcases/condition-flow-no-operators.yaml b/pkg/tmplexec/flow/testcases/condition-flow-no-operators.yaml
index 8cb687b248..4fef4b003c 100644
--- a/pkg/tmplexec/flow/testcases/condition-flow-no-operators.yaml
+++ b/pkg/tmplexec/flow/testcases/condition-flow-no-operators.yaml
@@ -1,23 +1,21 @@
-id: ghost-blog-detection
+id: condition-flow-no-operators
info:
- name: Ghost blog detection
+ name: Condition Flow No Operators
author: pdteam
severity: info
-
flow: dns() && http()
-
dns:
- name: "{{FQDN}}"
type: CNAME
-
+
http:
- method: GET
path:
- - "{{BaseURL}}?ref={{dns_cname}}"
+ - "{{BaseURL}}/?ref={{dns_cname}}"
matchers:
- type: word
words:
- - "ghost.io"
\ No newline at end of file
+ - "html>"
\ No newline at end of file
diff --git a/pkg/tmplexec/flow/testcases/condition-flow.yaml b/pkg/tmplexec/flow/testcases/condition-flow.yaml
index d1e2cbf9d2..ba837ab9ca 100644
--- a/pkg/tmplexec/flow/testcases/condition-flow.yaml
+++ b/pkg/tmplexec/flow/testcases/condition-flow.yaml
@@ -1,6 +1,6 @@
-id: ghost-blog-detection
+id: vercel-hosted-detection
info:
- name: Ghost blog detection
+ name: Vercel-hosted detection
author: pdteam
severity: info
@@ -14,14 +14,14 @@ dns:
matchers:
- type: word
words:
- - "ghost.io"
+ - "vercel-dns"
http:
- method: GET
path:
- - "{{BaseURL}}"
+ - "{{dns_cname}}"
matchers:
- type: word
words:
- - "ghost.io"
\ No newline at end of file
+ - "DEPLOYMENT_NOT_FOUND"
\ No newline at end of file
diff --git a/pkg/tmplexec/generic/exec.go b/pkg/tmplexec/generic/exec.go
index 4326141b15..f4da02f43a 100644
--- a/pkg/tmplexec/generic/exec.go
+++ b/pkg/tmplexec/generic/exec.go
@@ -85,7 +85,7 @@ func (g *Generic) ExecuteWithResults(ctx *scan.ScanContext) error {
if err != nil {
ctx.LogError(err)
if g.options.HostErrorsCache != nil {
- g.options.HostErrorsCache.MarkFailed(ctx.Input.MetaInput.ID(), err)
+ g.options.HostErrorsCache.MarkFailed(g.options.ProtocolType.String(), ctx.Input, err)
}
gologger.Warning().Msgf("[%s] Could not execute request for %s: %s\n", g.options.TemplateID, ctx.Input.MetaInput.PrettyPrint(), err)
}
diff --git a/pkg/tmplexec/interface.go b/pkg/tmplexec/interface.go
index f452789bd2..8871c6588e 100644
--- a/pkg/tmplexec/interface.go
+++ b/pkg/tmplexec/interface.go
@@ -1,10 +1,22 @@
package tmplexec
import (
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/scan"
"github.com/Explorer1092/nuclei/v3/pkg/tmplexec/flow"
"github.com/Explorer1092/nuclei/v3/pkg/tmplexec/generic"
"github.com/Explorer1092/nuclei/v3/pkg/tmplexec/multiproto"
+=======
+ "errors"
+ "regexp"
+ "strings"
+
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/flow"
+ "github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/generic"
+ "github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/multiproto"
+ "github.com/projectdiscovery/utils/errkit"
+>>>>>>> projectdiscovery-main
)
var (
@@ -30,3 +42,37 @@ type TemplateEngine interface {
// Name returns name of template engine
Name() string
}
+
+var (
+ // A temporary fix to remove errKind from error message
+ // this is because errkit is not used everywhere yet
+ reNoKind = regexp.MustCompile(`([\[][^][]+[\]]|errKind=[^ ]+) `)
+)
+
+// parseScanError parses given scan error and only returning the cause
+// instead of inefficient one
+func parseScanError(msg string) string {
+ if msg == "" {
+ return ""
+ }
+ if strings.HasPrefix(msg, "ReadStatusLine:") {
+ // last index is actual error (from rawhttp)
+ parts := strings.Split(msg, ":")
+ msg = strings.TrimSpace(parts[len(parts)-1])
+ }
+ if strings.Contains(msg, "read ") {
+ // same here
+ parts := strings.Split(msg, ":")
+ msg = strings.TrimSpace(parts[len(parts)-1])
+ }
+ e := errkit.FromError(errors.New(msg))
+ for _, err := range e.Errors() {
+ if err != nil && strings.Contains(err.Error(), "context deadline exceeded") {
+ continue
+ }
+ msg = reNoKind.ReplaceAllString(err.Error(), "")
+ return msg
+ }
+ wrapped := errkit.Append(errkit.New("failed to get error cause"), e).Error()
+ return reNoKind.ReplaceAllString(wrapped, "")
+}
diff --git a/pkg/tmplexec/multiproto/multi.go b/pkg/tmplexec/multiproto/multi.go
index 6fae261c06..7e9a021909 100644
--- a/pkg/tmplexec/multiproto/multi.go
+++ b/pkg/tmplexec/multiproto/multi.go
@@ -4,10 +4,19 @@ import (
"strconv"
"sync/atomic"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/output"
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
"github.com/Explorer1092/nuclei/v3/pkg/protocols/common/generators"
"github.com/Explorer1092/nuclei/v3/pkg/scan"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/output"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ stringsutil "github.com/projectdiscovery/utils/strings"
+>>>>>>> projectdiscovery-main
)
// Mutliprotocol is a template executer engine that executes multiple protocols
@@ -107,12 +116,26 @@ func (m *MultiProtocol) ExecuteWithResults(ctx *scan.ScanContext) error {
return ctx.Context().Err()
default:
}
-
- values := m.options.GetTemplateCtx(ctx.Input.MetaInput).GetAll()
- err := req.ExecuteWithResults(ctx.Input, output.InternalEvent(values), nil, multiProtoCallback)
- // if error skip execution of next protocols
+ inputItem := ctx.Input.Clone()
+ if m.options.InputHelper != nil && ctx.Input.MetaInput.Input != "" {
+ if inputItem.MetaInput.Input = m.options.InputHelper.Transform(inputItem.MetaInput.Input, req.Type()); inputItem.MetaInput.Input == "" {
+ return nil
+ }
+ }
+ // FIXME: this hack of using hash to get templateCtx has known issues scan context based approach should be adopted ASAP
+ values := m.options.GetTemplateCtx(inputItem.MetaInput).GetAll()
+ err := req.ExecuteWithResults(inputItem, output.InternalEvent(values), nil, multiProtoCallback)
+ // in case of fatal error skip execution of next protocols
if err != nil {
+ // always log errors
ctx.LogError(err)
+ // for some classes of protocols (i.e ssl) errors like tls handshake are a legitimate behavior so we don't stop execution
+ // connection failures are already tracked by the internal host error cache
+ // we use strings comparison as the error is not formalized into instance within the standard library
+ // within a flow instead we consider ssl errors as fatal, since a specific logic was requested
+ if req.Type() == types.SSLProtocol && stringsutil.ContainsAnyI(err.Error(), "protocol version not supported", "could not do tls handshake") {
+ continue
+ }
return err
}
}
diff --git a/pkg/tmplexec/multiproto/multi_test.go b/pkg/tmplexec/multiproto/multi_test.go
index feb6aee6fb..112b25e398 100644
--- a/pkg/tmplexec/multiproto/multi_test.go
+++ b/pkg/tmplexec/multiproto/multi_test.go
@@ -3,9 +3,11 @@ package multiproto_test
import (
"context"
"log"
+ "os"
"testing"
"time"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
"github.com/Explorer1092/nuclei/v3/pkg/catalog/disk"
"github.com/Explorer1092/nuclei/v3/pkg/loader/workflow"
@@ -15,6 +17,18 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/scan"
"github.com/Explorer1092/nuclei/v3/pkg/templates"
"github.com/Explorer1092/nuclei/v3/pkg/testutils"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
+ "github.com/projectdiscovery/nuclei/v3/pkg/input"
+ "github.com/projectdiscovery/nuclei/v3/pkg/loader/workflow"
+ "github.com/projectdiscovery/nuclei/v3/pkg/progress"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
+ "github.com/projectdiscovery/nuclei/v3/pkg/scan"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates"
+ "github.com/projectdiscovery/nuclei/v3/pkg/testutils"
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/ratelimit"
"github.com/stretchr/testify/require"
)
@@ -36,6 +50,7 @@ func setup() {
Catalog: disk.NewCatalog(config.DefaultConfig.TemplatesDirectory),
RateLimiter: ratelimit.New(context.Background(), uint(options.RateLimit), time.Second),
Parser: templates.NewParser(),
+ InputHelper: input.NewHelper(),
}
workflowLoader, err := workflow.NewLoader(&executerOpts)
if err != nil {
@@ -45,7 +60,6 @@ func setup() {
}
func TestMultiProtoWithDynamicExtractor(t *testing.T) {
- setup()
Template, err := templates.Parse("testcases/multiprotodynamic.yaml", nil, executerOpts)
require.Nil(t, err, "could not parse template")
@@ -54,7 +68,7 @@ func TestMultiProtoWithDynamicExtractor(t *testing.T) {
err = Template.Executer.Compile()
require.Nil(t, err, "could not compile template")
- input := contextargs.NewWithInput(context.Background(), "blog.projectdiscovery.io")
+ input := contextargs.NewWithInput(context.Background(), "http://scanme.sh")
ctx := scan.NewScanContext(context.Background(), input)
gotresults, err := Template.Executer.Execute(ctx)
require.Nil(t, err, "could not execute template")
@@ -62,7 +76,6 @@ func TestMultiProtoWithDynamicExtractor(t *testing.T) {
}
func TestMultiProtoWithProtoPrefix(t *testing.T) {
- setup()
Template, err := templates.Parse("testcases/multiprotowithprefix.yaml", nil, executerOpts)
require.Nil(t, err, "could not parse template")
@@ -71,9 +84,14 @@ func TestMultiProtoWithProtoPrefix(t *testing.T) {
err = Template.Executer.Compile()
require.Nil(t, err, "could not compile template")
- input := contextargs.NewWithInput(context.Background(), "blog.projectdiscovery.io")
+ input := contextargs.NewWithInput(context.Background(), "https://cloud.projectdiscovery.io/sign-in")
ctx := scan.NewScanContext(context.Background(), input)
gotresults, err := Template.Executer.Execute(ctx)
require.Nil(t, err, "could not execute template")
require.True(t, gotresults)
}
+
+func TestMain(m *testing.M) {
+ setup()
+ os.Exit(m.Run())
+}
diff --git a/pkg/tmplexec/multiproto/testcases/multiprotodynamic.yaml b/pkg/tmplexec/multiproto/testcases/multiprotodynamic.yaml
index edb20dfd34..7ffad43d7b 100644
--- a/pkg/tmplexec/multiproto/testcases/multiprotodynamic.yaml
+++ b/pkg/tmplexec/multiproto/testcases/multiprotodynamic.yaml
@@ -7,15 +7,7 @@ info:
dns:
- name: "{{FQDN}}" # DNS Request
- type: cname
-
- extractors:
- - type: dsl
- name: blogid
- dsl:
- - trim_suffix(cname,'.ghost.io')
- internal: true
-
+ type: a
http:
- method: GET # http request
@@ -25,6 +17,6 @@ http:
matchers:
- type: dsl
dsl:
- - contains(body,'ProjectDiscovery.io') # check for http string
- - blogid == 'projectdiscovery' # check for cname (extracted information from dns response)
+ - body == "ok"
+ - dns_a == '128.199.158.128' # check for A record (extracted information from dns response)
condition: and
\ No newline at end of file
diff --git a/pkg/tmplexec/multiproto/testcases/multiprotowithprefix.yaml b/pkg/tmplexec/multiproto/testcases/multiprotowithprefix.yaml
index 61dc410aed..e4dc241adc 100644
--- a/pkg/tmplexec/multiproto/testcases/multiprotowithprefix.yaml
+++ b/pkg/tmplexec/multiproto/testcases/multiprotowithprefix.yaml
@@ -20,7 +20,7 @@ http:
matchers:
- type: dsl
dsl:
- - contains(http_body,'ProjectDiscovery.io') # check for http string
- - trim_suffix(dns_cname,'.ghost.io') == 'projectdiscovery' # check for cname (extracted information from dns response)
- - ssl_subject_cn == 'blog.projectdiscovery.io'
+ - contains(http_body, 'ProjectDiscovery Cloud Platform') # check for http string
+ - dns_cname == 'cname.vercel-dns.com' # check for cname (extracted information from dns response)
+ - ssl_subject_cn == 'cloud.projectdiscovery.io'
condition: and
\ No newline at end of file
diff --git a/pkg/types/interfaces.go b/pkg/types/interfaces.go
index 4271ebf9c3..b922ccfc29 100644
--- a/pkg/types/interfaces.go
+++ b/pkg/types/interfaces.go
@@ -11,6 +11,14 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/asaskevich/govalidator"
+<<<<<<< HEAD
+=======
+<<<<<<< HEAD:v2/pkg/types/interfaces.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/types/interfaces.go
+>>>>>>> projectdiscovery-main
)
// JSONScalarToString converts an interface coming from json to string
diff --git a/pkg/types/nucleierr/kinds.go b/pkg/types/nucleierr/kinds.go
new file mode 100644
index 0000000000..9042beaade
--- /dev/null
+++ b/pkg/types/nucleierr/kinds.go
@@ -0,0 +1,28 @@
+package nucleierr
+
+import (
+ "strings"
+
+ "github.com/projectdiscovery/utils/errkit"
+)
+
+var (
+ // ErrTemplateLogic are errors that occured due to missing variable or something similar in template logic
+ // so this is more of a virtual error that is expected due to template logic
+ ErrTemplateLogic = errkit.NewPrimitiveErrKind("TemplateLogic", "Error expected due to template logic", isTemplateLogicKind)
+)
+
+// isTemplateLogicKind checks if an error is of template logic kind
+func isTemplateLogicKind(err *errkit.ErrorX) bool {
+ if err == nil || err.Cause() == nil {
+ return false
+ }
+ v := err.Cause().Error()
+ switch {
+ case strings.Contains(v, "timeout annotation deadline exceeded"):
+ return true
+ case strings.Contains(v, "stop execution due to unresolved variables"):
+ return true
+ }
+ return false
+}
diff --git a/pkg/types/resume.go b/pkg/types/resume.go
index 4be259f24b..7fb9839d1b 100644
--- a/pkg/types/resume.go
+++ b/pkg/types/resume.go
@@ -6,7 +6,15 @@ import (
"path/filepath"
"sync"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
+=======
+<<<<<<< HEAD:v2/pkg/types/resume.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/types/resume.go
+>>>>>>> projectdiscovery-main
"github.com/rs/xid"
)
diff --git a/pkg/types/types.go b/pkg/types/types.go
index 2b3783d8a2..d723d045a9 100644
--- a/pkg/types/types.go
+++ b/pkg/types/types.go
@@ -12,9 +12,23 @@ import (
"github.com/Explorer1092/nuclei/v3/pkg/model/types/severity"
"github.com/Explorer1092/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/goflags"
+<<<<<<< HEAD
errorutil "github.com/projectdiscovery/utils/errors"
+=======
+<<<<<<< HEAD:v2/pkg/types/types.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/severity"
+ "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+ errorutil "github.com/projectdiscovery/utils/errors"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/types/types.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
folderutil "github.com/projectdiscovery/utils/folder"
+ unitutils "github.com/projectdiscovery/utils/unit"
)
var (
@@ -22,6 +36,9 @@ var (
ErrNoMoreRequests = io.EOF
)
+// LoadHelperFileFunction can be used to load a helper file.
+type LoadHelperFileFunction func(helperFile, templatePath string, catalog catalog.Catalog) (io.ReadCloser, error)
+
// Options contains the configuration options for nuclei scanner.
type Options struct {
// Tags contains a list of tags to execute templates for. Multiple paths
@@ -202,6 +219,8 @@ type Options struct {
VerboseVerbose bool
// ShowVarDump displays variable dump
ShowVarDump bool
+ // VarDumpLimit limits the number of characters displayed in var dump
+ VarDumpLimit int
// No-Color disables the colored output.
NoColor bool
// UpdateTemplates updates the templates installed at startup (also used by cloud to update datasources)
@@ -219,6 +238,8 @@ type Options struct {
JSONExport string
// JSONLExport is the file to export JSONL output format to
JSONLExport string
+ // Redact redacts given keys in
+ Redact goflags.StringSlice
// EnableProgressBar enables progress bar
EnableProgressBar bool
// TemplateDisplay displays the template contents
@@ -277,8 +298,6 @@ type Options struct {
SNI string
// InputFileMode specifies the mode of input file (jsonl, burp, openapi, swagger, etc)
InputFileMode string
- // DialerTimeout sets the timeout for network requests.
- DialerTimeout time.Duration
// DialerKeepAlive sets the keep alive duration for network requests.
DialerKeepAlive time.Duration
// Interface to use for network scan
@@ -291,8 +310,6 @@ type Options struct {
ResponseReadSize int
// ResponseSaveSize is the maximum size of response to save
ResponseSaveSize int
- // ResponseReadTimeout is response read timeout in seconds
- ResponseReadTimeout time.Duration
// Health Check
HealthCheck bool
// Time to wait between each input read operation before closing the stream
@@ -363,6 +380,12 @@ type Options struct {
FuzzingMode string
// TlsImpersonate enables TLS impersonation
TlsImpersonate bool
+ // DisplayFuzzPoints enables display of fuzz points for fuzzing
+ DisplayFuzzPoints bool
+ // FuzzAggressionLevel is the level of fuzzing aggression (low, medium, high.)
+ FuzzAggressionLevel string
+ // FuzzParamFrequency is the frequency of fuzzing parameters
+ FuzzParamFrequency int
// CodeTemplateSignaturePublicKey is the custom public key used to verify the template signature (algorithm is automatically inferred from the length)
CodeTemplateSignaturePublicKey string
// CodeTemplateSignatureAlgorithm specifies the sign algorithm (rsa, ecdsa)
@@ -373,12 +396,20 @@ type Options struct {
EnableCodeTemplates bool
// DisableUnsignedTemplates disables processing of unsigned templates
DisableUnsignedTemplates bool
+ // EnableSelfContainedTemplates disables processing of self-contained templates
+ EnableSelfContainedTemplates bool
+ // EnableFileTemplates enables file templates
+ EnableFileTemplates bool
// Disables cloud upload
EnableCloudUpload bool
// ScanID is the scan ID to use for cloud upload
ScanID string
// ScanName is the name of the scan to be uploaded
ScanName string
+ // ScanUploadFile is the jsonl file to upload scan results to cloud
+ ScanUploadFile string
+ // TeamID is the team ID to use for cloud upload
+ TeamID string
// JsConcurrency is the number of concurrent js routines to run
JsConcurrency int
// SecretsFile is file containing secrets for nuclei
@@ -399,6 +430,82 @@ type Options struct {
HttpApiEndpoint string
// ListTemplateProfiles lists all available template profiles
ListTemplateProfiles bool
+ // LoadHelperFileFunction is a function that will be used to execute LoadHelperFile.
+ // If none is provided, then the default implementation will be used.
+ LoadHelperFileFunction LoadHelperFileFunction
+ // timeouts contains various types of timeouts used in nuclei
+ // these timeouts are derived from dial-timeout (-timeout) with known multipliers
+ // This is internally managed and does not need to be set by user by explicitly setting
+ // this overrides the default/derived one
+ timeouts *Timeouts
+}
+
+// SetTimeouts sets the timeout variants to use for the executor
+func (opts *Options) SetTimeouts(t *Timeouts) {
+ opts.timeouts = t
+}
+
+// GetTimeouts returns the timeout variants to use for the executor
+func (eo *Options) GetTimeouts() *Timeouts {
+ if eo.timeouts != nil {
+ // redundant but apply to avoid any potential issues
+ eo.timeouts.ApplyDefaults()
+ return eo.timeouts
+ }
+ // set timeout variant value
+ eo.timeouts = NewTimeoutVariant(eo.Timeout)
+ eo.timeouts.ApplyDefaults()
+ return eo.timeouts
+}
+
+// Timeouts is a struct that contains all the timeout variants for nuclei
+// dialer timeout is used to derive other timeouts
+type Timeouts struct {
+ // DialTimeout for fastdialer (default 10s)
+ DialTimeout time.Duration
+ // Tcp(Network Protocol) Read From Connection Timeout (default 5s)
+ TcpReadTimeout time.Duration
+ // Http Response Header Timeout (default 10s)
+ // this timeout prevents infinite hangs started by server if any
+ // this is temporarily overridden when using @timeout request annotation
+ HttpResponseHeaderTimeout time.Duration
+ // HttpTimeout for http client (default -> 3 x dial-timeout = 30s)
+ HttpTimeout time.Duration
+ // JsCompilerExec timeout/deadline (default -> 2 x dial-timeout = 20s)
+ JsCompilerExecutionTimeout time.Duration
+ // CodeExecutionTimeout for code execution (default -> 3 x dial-timeout = 30s)
+ CodeExecutionTimeout time.Duration
+}
+
+// NewTimeoutVariant creates a new timeout variant with the given dial timeout in seconds
+func NewTimeoutVariant(dialTimeoutSec int) *Timeouts {
+ tv := &Timeouts{
+ DialTimeout: time.Duration(dialTimeoutSec) * time.Second,
+ }
+ tv.ApplyDefaults()
+ return tv
+}
+
+// ApplyDefaults applies default values to timeout variants when missing
+func (tv *Timeouts) ApplyDefaults() {
+ if tv.DialTimeout == 0 {
+ tv.DialTimeout = 10 * time.Second
+ }
+ if tv.TcpReadTimeout == 0 {
+ tv.TcpReadTimeout = 5 * time.Second
+ }
+ if tv.HttpResponseHeaderTimeout == 0 {
+ tv.HttpResponseHeaderTimeout = 10 * time.Second
+ }
+ if tv.HttpTimeout == 0 {
+ tv.HttpTimeout = 3 * tv.DialTimeout
+ }
+ if tv.JsCompilerExecutionTimeout == 0 {
+ tv.JsCompilerExecutionTimeout = 2 * tv.DialTimeout
+ }
+ if tv.CodeExecutionTimeout == 0 {
+ tv.CodeExecutionTimeout = 3 * tv.DialTimeout
+ }
}
// ShouldLoadResume resume file
@@ -435,9 +542,8 @@ func DefaultOptions() *Options {
Timeout: 5,
Retries: 1,
MaxHostError: 30,
- ResponseReadSize: 10 * 1024 * 1024,
- ResponseSaveSize: 1024 * 1024,
- ResponseReadTimeout: 5 * time.Second,
+ ResponseReadSize: 10 * unitutils.Mega,
+ ResponseSaveSize: unitutils.Mega,
}
}
@@ -459,10 +565,21 @@ func (options *Options) ParseHeadlessOptionalArguments() map[string]string {
return optionalArguments
}
-// LoadHelperFile loads a helper file needed for the template
+// LoadHelperFile loads a helper file needed for the template.
+//
+// If LoadHelperFileFunction is set, then that function will be used.
+// Otherwise, the default implementation will be used, which respects the sandbox rules and only loads files from allowed directories.
+func (options *Options) LoadHelperFile(helperFile, templatePath string, catalog catalog.Catalog) (io.ReadCloser, error) {
+ if options.LoadHelperFileFunction != nil {
+ return options.LoadHelperFileFunction(helperFile, templatePath, catalog)
+ }
+ return options.defaultLoadHelperFile(helperFile, templatePath, catalog)
+}
+
+// defaultLoadHelperFile loads a helper file needed for the template
// this respects the sandbox rules and only loads files from
// allowed directories
-func (options *Options) LoadHelperFile(helperFile, templatePath string, catalog catalog.Catalog) (io.ReadCloser, error) {
+func (options *Options) defaultLoadHelperFile(helperFile, templatePath string, catalog catalog.Catalog) (io.ReadCloser, error) {
if !options.AllowLocalFileAccess {
// if global file access is disabled try loading with restrictions
absPath, err := options.GetValidAbsPath(helperFile, templatePath)
@@ -517,7 +634,7 @@ func (o *Options) GetValidAbsPath(helperFilePath, templatePath string) (string,
return "", errorutil.New("access to helper file %v denied", helperFilePath)
}
-// isRootDir checks if given is root directory
+// isHomeDir checks if given is home directory
func isHomeDir(path string) bool {
homeDir := folderutil.HomeDirOrDefault("")
return strings.HasPrefix(path, homeDir)
diff --git a/pkg/utils/http_probe.go b/pkg/utils/http_probe.go
index fcdf413758..ff5e83d5b3 100644
--- a/pkg/utils/http_probe.go
+++ b/pkg/utils/http_probe.go
@@ -29,6 +29,7 @@ func ProbeURL(input string, httpxclient *httpx.HTTPX) string {
if _, err = httpxclient.Do(req, httpx.UnsafeOptions{}); err != nil {
continue
}
+
return formedURL
}
return ""
diff --git a/pkg/utils/monitor/monitor.go b/pkg/utils/monitor/monitor.go
index 6441fdf20d..cefb9506b8 100644
--- a/pkg/utils/monitor/monitor.go
+++ b/pkg/utils/monitor/monitor.go
@@ -16,6 +16,7 @@ import (
"github.com/DataDog/gostackparse"
"github.com/projectdiscovery/gologger"
permissionutil "github.com/projectdiscovery/utils/permission"
+ unitutils "github.com/projectdiscovery/utils/unit"
"github.com/rs/xid"
)
@@ -118,7 +119,7 @@ func (s *Agent) monitorWorker(cancel context.CancelFunc) {
// getStack returns full stack trace of the program
var getStack = func(all bool) []byte {
- for i := 1024 * 1024; ; i *= 2 {
+ for i := unitutils.Mega; ; i *= 2 {
buf := make([]byte, i)
if n := runtime.Stack(buf, all); n < i {
return buf[:n-1]
diff --git a/pkg/utils/stats/stats.go b/pkg/utils/stats/stats.go
index 590608d1e2..f1edf2ea33 100644
--- a/pkg/utils/stats/stats.go
+++ b/pkg/utils/stats/stats.go
@@ -2,23 +2,22 @@ package stats
import (
"fmt"
- "sync"
"sync/atomic"
"github.com/logrusorgru/aurora"
"github.com/projectdiscovery/gologger"
+ mapsutil "github.com/projectdiscovery/utils/maps"
)
// Storage is a storage for storing statistics information
// about the nuclei engine displaying it at user-defined intervals.
type Storage struct {
- data map[string]*storageDataItem
- mutex *sync.RWMutex
+ data *mapsutil.SyncLockMap[string, *storageDataItem]
}
type storageDataItem struct {
description string
- value int64
+ value atomic.Int64
}
var Default *Storage
@@ -59,38 +58,32 @@ func GetValue(name string) int64 {
// New creates a new storage object
func New() *Storage {
- return &Storage{data: make(map[string]*storageDataItem), mutex: &sync.RWMutex{}}
+ data := mapsutil.NewSyncLockMap[string, *storageDataItem]()
+ return &Storage{data: data}
}
// NewEntry creates a new entry in the storage object
func (s *Storage) NewEntry(name, description string) {
- s.mutex.Lock()
- s.data[name] = &storageDataItem{description: description, value: 0}
- s.mutex.Unlock()
+ _ = s.data.Set(name, &storageDataItem{description: description, value: atomic.Int64{}})
}
// Increment increments the value for a name string
func (s *Storage) Increment(name string) {
- s.mutex.RLock()
- data, ok := s.data[name]
- s.mutex.RUnlock()
+ data, ok := s.data.Get(name)
if !ok {
return
}
-
- atomic.AddInt64(&data.value, 1)
+ data.value.Add(1)
}
// Display displays the stats for a name
func (s *Storage) Display(name string) {
- s.mutex.RLock()
- data, ok := s.data[name]
- s.mutex.RUnlock()
+ data, ok := s.data.Get(name)
if !ok {
return
}
- dataValue := atomic.LoadInt64(&data.value)
+ dataValue := data.value.Load()
if dataValue == 0 {
return // don't show for nil stats
}
@@ -98,14 +91,12 @@ func (s *Storage) Display(name string) {
}
func (s *Storage) DisplayAsWarning(name string) {
- s.mutex.RLock()
- data, ok := s.data[name]
- s.mutex.RUnlock()
+ data, ok := s.data.Get(name)
if !ok {
return
}
- dataValue := atomic.LoadInt64(&data.value)
+ dataValue := data.value.Load()
if dataValue == 0 {
return // don't show for nil stats
}
@@ -115,14 +106,12 @@ func (s *Storage) DisplayAsWarning(name string) {
// ForceDisplayWarning forces the display of a warning
// regardless of current verbosity level
func (s *Storage) ForceDisplayWarning(name string) {
- s.mutex.RLock()
- data, ok := s.data[name]
- s.mutex.RUnlock()
+ data, ok := s.data.Get(name)
if !ok {
return
}
- dataValue := atomic.LoadInt64(&data.value)
+ dataValue := data.value.Load()
if dataValue == 0 {
return // don't show for nil stats
}
@@ -131,13 +120,10 @@ func (s *Storage) ForceDisplayWarning(name string) {
// GetValue returns the value for a set variable
func (s *Storage) GetValue(name string) int64 {
- s.mutex.RLock()
- data, ok := s.data[name]
- s.mutex.RUnlock()
+ data, ok := s.data.Get(name)
if !ok {
return 0
}
- dataValue := atomic.LoadInt64(&data.value)
- return dataValue
+ return data.value.Load()
}
diff --git a/pkg/utils/template_path.go b/pkg/utils/template_path.go
index 833284f943..686f97022b 100644
--- a/pkg/utils/template_path.go
+++ b/pkg/utils/template_path.go
@@ -3,7 +3,12 @@ package utils
import (
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog/config"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
+ "github.com/projectdiscovery/nuclei/v3/pkg/keys"
+>>>>>>> projectdiscovery-main
)
const (
@@ -12,16 +17,13 @@ const (
)
// TemplatePathURL returns the Path and URL for the provided template
-func TemplatePathURL(fullPath, templateId string) (string, string) {
- var templateDirectory string
+func TemplatePathURL(fullPath, templateId, templateVerifier string) (path string, url string) {
configData := config.DefaultConfig
if configData.TemplatesDirectory != "" && strings.HasPrefix(fullPath, configData.TemplatesDirectory) {
- templateDirectory = configData.TemplatesDirectory
- } else {
- return "", ""
+ path = strings.TrimPrefix(strings.TrimPrefix(fullPath, configData.TemplatesDirectory), "/")
}
-
- finalPath := strings.TrimPrefix(strings.TrimPrefix(fullPath, templateDirectory), "/")
- templateURL := TemplatesRepoURL + templateId
- return finalPath, templateURL
+ if templateVerifier == keys.PDVerifier {
+ url = TemplatesRepoURL + templateId
+ }
+ return
}
diff --git a/pkg/utils/utils.go b/pkg/utils/utils.go
index 4ab9f6acdc..afb0f070ba 100644
--- a/pkg/utils/utils.go
+++ b/pkg/utils/utils.go
@@ -2,12 +2,26 @@ package utils
import (
"errors"
+ "fmt"
"io"
"net/url"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/catalog"
+=======
+<<<<<<< HEAD:v2/pkg/utils/utils.go
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog"
+ "github.com/Explorer1092/nuclei/v2/pkg/catalog/config"
+ "github.com/Explorer1092/nuclei/v2/pkg/utils/yaml"
+=======
+ "github.com/cespare/xxhash"
+ "github.com/projectdiscovery/nuclei/v3/pkg/catalog"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/utils/utils.go
+>>>>>>> projectdiscovery-main
"github.com/projectdiscovery/retryablehttp-go"
+ mapsutil "github.com/projectdiscovery/utils/maps"
+ "golang.org/x/exp/constraints"
)
func IsBlank(value string) bool {
@@ -31,7 +45,7 @@ func IsURL(input string) bool {
return err == nil && u.Scheme != "" && u.Host != ""
}
-// ReadFromPathOrURL reads and returns the contents of a file or url.
+// ReaderFromPathOrURL reads and returns the contents of a file or url.
func ReaderFromPathOrURL(templatePath string, catalog catalog.Catalog) (io.ReadCloser, error) {
if IsURL(templatePath) {
resp, err := retryablehttp.DefaultClient().Get(templatePath)
@@ -57,3 +71,13 @@ func StringSliceContains(slice []string, item string) bool {
}
return false
}
+
+// MapHash generates a hash for any give map
+func MapHash[K constraints.Ordered, V any](m map[K]V) uint64 {
+ keys := mapsutil.GetSortedKeys(m)
+ var sb strings.Builder
+ for _, k := range keys {
+ sb.WriteString(fmt.Sprintf("%v:%v\n", k, m[k]))
+ }
+ return xxhash.Sum64([]byte(sb.String()))
+}
diff --git a/pkg/utils/yaml/preprocess.go b/pkg/utils/yaml/preprocess.go
index 34af2e2cd3..b289c8ea31 100644
--- a/pkg/utils/yaml/preprocess.go
+++ b/pkg/utils/yaml/preprocess.go
@@ -7,7 +7,15 @@ import (
"regexp"
"strings"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/templates/extensions"
+=======
+<<<<<<< HEAD:v2/pkg/utils/yaml/preprocess.go
+ "github.com/Explorer1092/nuclei/v2/pkg/templates/extensions"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/templates/extensions"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/utils/yaml/preprocess.go
+>>>>>>> projectdiscovery-main
fileutil "github.com/projectdiscovery/utils/file"
stringsutil "github.com/projectdiscovery/utils/strings"
)
diff --git a/pkg/workflows/workflows.go b/pkg/workflows/workflows.go
index 1deb62e9c0..889c7e78f8 100644
--- a/pkg/workflows/workflows.go
+++ b/pkg/workflows/workflows.go
@@ -3,10 +3,24 @@ package workflows
import (
"fmt"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
"github.com/Explorer1092/nuclei/v3/pkg/protocols"
templateTypes "github.com/Explorer1092/nuclei/v3/pkg/templates/types"
+=======
+<<<<<<< HEAD:v2/pkg/workflows/workflows.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+ "github.com/Explorer1092/nuclei/v2/pkg/protocols"
+ templateTypes "github.com/Explorer1092/nuclei/v2/pkg/templates/types"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+ "github.com/projectdiscovery/nuclei/v3/pkg/protocols"
+ templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/workflows/workflows.go
+>>>>>>> projectdiscovery-main
)
// Workflow is a workflow to execute with chained requests, etc.
diff --git a/pkg/workflows/workflows_test.go b/pkg/workflows/workflows_test.go
index 2d5f9ae5ca..15b1cc976c 100644
--- a/pkg/workflows/workflows_test.go
+++ b/pkg/workflows/workflows_test.go
@@ -3,8 +3,18 @@ package workflows
import (
"testing"
+<<<<<<< HEAD
"github.com/Explorer1092/nuclei/v3/pkg/model/types/stringslice"
"github.com/Explorer1092/nuclei/v3/pkg/operators"
+=======
+<<<<<<< HEAD:v2/pkg/workflows/workflows_test.go
+ "github.com/Explorer1092/nuclei/v2/pkg/model/types/stringslice"
+ "github.com/Explorer1092/nuclei/v2/pkg/operators"
+=======
+ "github.com/projectdiscovery/nuclei/v3/pkg/model/types/stringslice"
+ "github.com/projectdiscovery/nuclei/v3/pkg/operators"
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a:pkg/workflows/workflows_test.go
+>>>>>>> projectdiscovery-main
"github.com/stretchr/testify/require"
)
diff --git a/static/nuclei-cover-image.png b/static/nuclei-cover-image.png
new file mode 100644
index 0000000000..caab6fb2d4
Binary files /dev/null and b/static/nuclei-cover-image.png differ
diff --git a/static/nuclei-cover.png b/static/nuclei-cover.png
new file mode 100644
index 0000000000..caab6fb2d4
Binary files /dev/null and b/static/nuclei-cover.png differ
diff --git a/static/nuclei-getting-started.png b/static/nuclei-getting-started.png
new file mode 100644
index 0000000000..9d57eae784
Binary files /dev/null and b/static/nuclei-getting-started.png differ
diff --git a/static/nuclei-template-example.png b/static/nuclei-template-example.png
new file mode 100644
index 0000000000..22d71daaf8
Binary files /dev/null and b/static/nuclei-template-example.png differ
diff --git a/static/nuclei-templates-teamcity-example.png b/static/nuclei-templates-teamcity-example.png
new file mode 100644
index 0000000000..83fc404c16
Binary files /dev/null and b/static/nuclei-templates-teamcity-example.png differ
diff --git a/static/nuclei-templates-teamcity.png b/static/nuclei-templates-teamcity.png
new file mode 100644
index 0000000000..04d1581267
Binary files /dev/null and b/static/nuclei-templates-teamcity.png differ
diff --git a/static/nuclei-write-your-first-template.png b/static/nuclei-write-your-first-template.png
new file mode 100644
index 0000000000..0c352d7062
Binary files /dev/null and b/static/nuclei-write-your-first-template.png differ
diff --git a/static/projectdiscovery-browse-results.gif b/static/projectdiscovery-browse-results.gif
new file mode 100644
index 0000000000..7dced0f3b3
Binary files /dev/null and b/static/projectdiscovery-browse-results.gif differ
diff --git a/static/teamcity-example.png b/static/teamcity-example.png
new file mode 100644
index 0000000000..f70b084b24
Binary files /dev/null and b/static/teamcity-example.png differ
+
+
+
+
+
+
+---
-
diff --git a/README_CN.md b/README_CN.md
index 09d24eeed4..6ec13b116f 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -32,7 +32,8 @@
English •
中文 •
Korean •
- Indonesia
+ Indonesia •
+ Spanish
---
@@ -60,7 +61,15 @@ Nuclei使用零误报的定制模板向目标发送请求，同时可以对主
Nuclei需要 **go1.21** 才能安装成功。执行下列命令安装最新版本的Nuclei
```sh
+<<<<<<< HEAD
go install -v github.com/Explorer1092/nuclei/v3/cmd/nuclei@latest
+=======
+<<<<<<< HEAD
+go install -v github.com/Explorer1092/nuclei/v2/cmd/nuclei@latest
+=======
+go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
+>>>>>>> 419f08f61ce5ca2d3f0eae9fe36dc7c44c1f532a
+>>>>>>> projectdiscovery-main
```
