Details
| ID | WebScraper |
|---|---|
| Language | python |
| From Version | 6.0.0 |
| Docker Image | demisto/bs4-py3:1.0.0.10120494 |
README
An Automation Script to Web Scrap a URL or HTML Page
Script Data
| Name | Description |
|---|---|
| Script Type | python3 |
| Tags | |
| Cortex XSOAR Version | 6.0.0 |
Inputs
| Argument Name | Description |
|---|---|
| page_url | Page url to scrap |
| page_html | HTML page to scrap |
| headers | Request headers |
| params | Request parameters |
| navigator_tree | HTML tags navigation tree, example: “body.table” |
| insecure | Ignore certificate validation errors |
Outputs
| Path | Description | Type |
|---|---|---|
| WebScraper.Tree | Scraped Pages | String |
Script Example
!WebScraper page_url=https://example.com navigator_tree=body
Context Example
{
"WebScraper": {
"Tree": [
{
"h1": "Example Domain"
},
{
"p": [
"This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.",
{
"a": {
"#text": "More information...",
"@href": "https://www.iana.org/domains/example"
}
}
]
}
]
}
}
Human Readable Output
Scrapping completed!
args: - description: Page url to scrap. name: page_url predefined: - '' - description: HTML page to scrap. name: page_html predefined: - '' - description: Request headers. name: headers - description: Request parameters. name: params - description: 'HTML tags navigation tree, example: "body.table".' name: navigator_tree - defaultValue: 'False' description: Ignore certificate validation errors. name: insecure comment: An Automation Script to Web Scrap a URL or HTML Page. commonfields: id: WebScraper version: -1 name: WebScraper outputs: - contextPath: WebScraper.Tree description: Scraped Pages. type: String script: '-' subtype: python3 tags: - '' timeout: '0' type: python dockerimage: demisto/bs4-py3:1.0.0.10120494 fromversion: 6.0.0 tests: - No tests