\"\"\" soup = BeautifulSoup(html_doc, 'lxml') # Using the lxml parser # Get all text, stripping tags all_text = soup.get_text(separator=' ', strip=True) print(\"All Text (including boilerplate):\") print(all_text) # Output: Sample Page Main Title Home About This is the primary article # content we want to keep. It discusses important topics. Another # paragraph of useful information. console.log('Some script'); Copyright # 2023. Some footer links. # Attempt to get only main content text (simple example) main_content_tag = soup.find('main') if main_content_tag: main_text = main_content_tag.get_text(separator=' ', strip=True) print(\"\\nMain Content Text (simple extraction):\") print(main_text) # Output: This is the primary article content we want to keep. It # discusses important topics. Another paragraph of useful # information. console.log('Some script'); else: print(\"\\n'main' tag not found.\")As the example shows, simply calling get_text() on the whole document often includes unwanted text from headers, footers, and potentially scripts if they contain text nodes. While finding specific tags like

can help, this relies on semantic HTML usage, which isn't always consistent across websites. Notice also that the simple extraction above still included the content of the

\"\"\" soup = BeautifulSoup(html_doc, 'lxml') # Tags to remove tags_to_remove = ['script', 'style', 'nav', 'header', 'footer', 'aside'] # Also remove elements identified by common boilerplate classes/ids (example) selectors_to_remove = ['.sidebar', '#ads'] for tag_name in tags_to_remove: for tag in soup.find_all(tag_name): tag.decompose() # Remove the tag and its contents for selector in selectors_to_remove: for tag in soup.select(selector): # Use CSS selectors tag.decompose() # Extract text from the modified soup cleaned_text = soup.get_text(separator=' ', strip=True) print(\"Text after Rule-Based Removal:\") print(cleaned_text) # Output: Sample Page This is the primary article content we want to keep. # It discusses important topics. Another paragraph of useful information.While effective for removing obvious boilerplate, rule-based methods have limitations. They can be brittle; websites structure content differently, and relying on specific tags like

or classes like sidebar isn't universally reliable. Aggressive removal might also discard useful information if, for instance, a figure caption is placed within a