大數(shù)據(jù)系統(tǒng)工程實(shí)訓(xùn)
定 價(jià)：59 元
- 作者：張志偉
- 出版時(shí)間：2025/9/1
- ISBN：9787121516702
- 出 版 社：電子工業(yè)出版社
- 中圖法分類：TP274
- 頁碼：244
- 紙張：
- 版次：01
- 開本：16開
大數(shù)據(jù)技術(shù)的落地依賴“工具鏈掌握+場景化應(yīng)用”的雙重能力。從 Python 爬蟲、Hive數(shù)據(jù)分析到 Flink 實(shí)時(shí)計(jì)算、數(shù)倉架構(gòu)設(shè)計(jì)，技能點(diǎn)的綜合應(yīng)用能力已成為企業(yè)招聘的核心標(biāo)準(zhǔn)。本書以“真實(shí)項(xiàng)目驅(qū)動實(shí)訓(xùn)”為核心思路，精選 4 個(gè)典型實(shí)訓(xùn)項(xiàng)目構(gòu)建階梯式訓(xùn)練體系，涵蓋離線處理、實(shí)時(shí)計(jì)算、數(shù)倉設(shè)計(jì)等核心場景，強(qiáng)化工程思維；整合 Python 爬蟲、Hive、Flink、Kafka 等多種主流工具，覆蓋數(shù)據(jù)采集、清洗、存儲、分析、可視化全流程；融入大數(shù)據(jù)競賽考點(diǎn)，銜接崗位技能需求。本書適合作為高等學(xué)校大數(shù)據(jù)相關(guān)專業(yè)的實(shí)訓(xùn)教材，也可為數(shù)據(jù)工程從業(yè)者提供實(shí)踐參考。
張志偉，副教授，宿州學(xué)院信息工程學(xué)院軟件工程教研室主任，博士畢業(yè)于華南理工大學(xué)計(jì)算機(jī)科學(xué)與技術(shù)專業(yè)，研究方向?yàn)閿?shù)據(jù)科學(xué)與大數(shù)據(jù)技術(shù)、人工智能，主持多項(xiàng)國家自然科學(xué)基金委員會項(xiàng)目和省級項(xiàng)目，編寫圖書3部。
第 1 章 歷史天氣數(shù)據(jù)分析項(xiàng)目································································································.1
任務(wù)一 需求分析·················································································································.1
任務(wù)二 技術(shù)架構(gòu)分析及設(shè)計(jì) ·····························································································.2
任務(wù)三 歷史天氣數(shù)據(jù)采集 ·································································································.5
任務(wù)四 導(dǎo)入天氣數(shù)據(jù)至 Hive···························································································.13
任務(wù)五 歷史天氣數(shù)據(jù)分析 ·······························································································.22
任務(wù)六 結(jié)果指標(biāo)表導(dǎo)出···································································································.33
任務(wù)七 數(shù)據(jù)可視化···········································································································.36
第 2 章 音樂推薦系統(tǒng)··············································································································.44
任務(wù)一 需求分析···············································································································.44
任務(wù)二 技術(shù)架構(gòu)分析及設(shè)計(jì) ···························································································.45
任務(wù)三 數(shù)據(jù)集合和項(xiàng)目概述 ···························································································.47
任務(wù)四 數(shù)據(jù)加載模塊·······································································································.52
任務(wù)五 數(shù)據(jù)統(tǒng)計(jì)模塊·······································································································.55
任務(wù)六 離線推薦模塊·······································································································.59
任務(wù)七 實(shí)時(shí)推薦模塊·······································································································.65
第 3 章 電商離線數(shù)倉··············································································································.72
任務(wù)一 需求分析···············································································································.72
任務(wù)二 數(shù)倉概述及架構(gòu)分析 ···························································································.73
任務(wù)三 數(shù)據(jù)源···················································································································.75
任務(wù)四 數(shù)倉建設(shè)···············································································································.77
任務(wù)五 工作流調(diào)度···········································································································117
任務(wù)六 數(shù)據(jù)可視化·········································································································.128
第 4 章 智慧社區(qū)實(shí)時(shí)數(shù)倉····································································································.136
任務(wù)一 需求分析·············································································································.136
任務(wù)二 技術(shù)架構(gòu)分析及設(shè)計(jì) ·························································································.137
任務(wù)三 數(shù)據(jù)源與預(yù)處理·································································································.140
任務(wù)四 實(shí)時(shí)計(jì)算框架配置 ·····························································································.153
任務(wù)五 DIM 層構(gòu)建········································································································.155
任務(wù)六 ODS 層構(gòu)建········································································································.169
任務(wù)七 DWD 層構(gòu)建 ······································································································.174
任務(wù)八 DWS 層構(gòu)建·······································································································.182
任務(wù)九 數(shù)據(jù)可視化與應(yīng)用 ·····························································································.192
附錄 A Hadoop 部署與配置··································································································.196
附錄 B MySQL 部署··············································································································.206
附錄 C Hive 部署與配置 ·······································································································.208
附錄 D DataX 部署與配置 ····································································································.215
附錄 E Zookeeper 部署與配置·····························································································.216
附錄 F Kafka 部署與配置······································································································.220
附錄 G Flume 部署與配置····································································································.224
附錄 H DolphinScheduler 部署與配置················································································.227
附錄 I Superset 部署與配置·································································································.234
書單推薦
